Picking out the vibe coders

To study this at scale, I built a classifier that assigns each GitHub account a rough probability of AI coding-tool adoption from public behavioural signals.

The training sample is 276 GitHub accounts: 74 confirmed AI adopters, identified through CLAUDE.md files or Co-Authored-By: Claude trailers, and 202 controls with active commit histories before and after late 2023 and no AI markers. The classifier uses 43 behavioural features: commit cadence, message length and structure, PR body patterns, and inter-commit timing. It explicitly excludes the labelling artefacts themselves.

The random forest model I ended up with correctly identifies the right outcome around 94% of the time. Remove all markers from the content (so it’s working purely from patterns of activity) and it still gets it right about 91% of the time.

Thankfully, Claude Code is not the only tool that leaves obvious traces. Aider does too. When I run the classifier on Aider users, a tool it was never trained on, it flags them as AI-tool users 73% of the time, compared to just 3% for people not using any AI coding tools.

So whatever the classifier is detecting, it is not just Claude-specific style. It looks like a broader “this developer’s behaviour looks AI-assisted” signal.

If you want the methodology in detail, the paper is here. For this post, the key point is simpler: I can assign an approximate AI-adoption score to public GitHub accounts, then ask whether higher adoption lines up with different activity metrics.

So what about countries?

So, we know at an individual level that AI users seem to work differently. But what about entire countries? If all of China has indeed gone OpenClaw-crazy, are they pushing endless agentic products? I was hoping this might give me a signal about what was going on.

It turns out it is just as puzzling.

With per-account AI scores in hand, I aggregated them by country. Take the 4,824 scored accounts, group them by country, and use the mean classifier score as a country-level AI adoption measure.

This is the weaker part of the analysis, and I want to be upfront about that. Country-level GitHub Archive data is noisy. The 2024 cross-section is thin. The adoption measure inherits whatever biases the classifier has. So I do not read this as clean evidence that AI adoption causes country-level productivity changes.

I read it as a warning sign about measurement.

For the dependent variables, I used a GitHub Archive panel from 2022 to 2024 with two common activity measures: commits per active developer and pull requests per active developer. If both were clean productivity metrics, you would expect them to broadly tell the same story.

They do not.

On the left: commits per developer in 2024, plotted against the country’s mean classifier-derived AI adoption score. Higher adoption, lower commits. The slope tilts down.

On the right: pull requests per developer, same countries, same year, same adoption measure. The slope is slightly positive and statistically indistinguishable from zero.

Same countries. Same year. Same adoption measure. Different activity metric, different story.

That is the point. I do not think the country result is strong enough to carry a claim about national productivity. But it is exactly what you would expect to see if commit counts are partly measuring workflow granularity rather than output.

The PR plot twist

So here is the bit that made me sit up.

The commit stuff measures workflow tempo: how often you hit save, how long you wait between commits, how chatty your messages are. That is interesting, but it is not the same as shipping work. To check that, I collected authored pull requests for the full classifier cohort and re-ran the before/after comparison using PR-specific outcomes.

Coverage. Of 276 accounts (74 treated, 202 controls), 275 had retrievable PR data. The treated group is more PR-active overall: 58 of 73 had at least one authored PR, versus 160 of 202 controls. But 15 treated accounts had zero PRs, and 13 hit the 300-PR retrieval cap. So the picture is mixed.

The volume story is strong. Confirmed adopters open +46.4 more PRs post-adoption and merge +42.6 more. On a per-month basis, that is +1.64 opened and +1.51 merged. All four numbers are significant at p < 0.001 and survive FDR correction across ten PR outcome tests. The effect also survives dropping capped accounts, dropping zero-PR accounts, and restricting to the highest-confidence adopters.

The rate story is messier. Merge rate rises by about 0.16 (p = 0.004), and median hours-to-merge falls by roughly 18.5 hours (p = 0.034). But when I restrict to accounts with PRs in both windows, the merge-rate effect collapses to basically nothing (+0.03, p = 0.39) while the volume effects stay significant.

What is happening? When an account had zero PRs in the pre-period, merge rate and merge time are mechanically set to zero. That artificially inflates the post-period gain. So I treat PR volume as the primary signal and merge rate as secondary, at best.

What this means for the puzzle. The country-level panel found PRs per developer effectively flat. The account-level extension finds confirmed adopters opening and merging substantially more PRs. The apparent contradiction is not a contradiction. It is a scale mismatch.

At the country level, the treatment is a small adoption share diluted across every developer in the country. At the account level, the treatment is confirmed adoption compared with confirmed non-adoption. A +46 PR effect among 10-15% of developers is invisible when averaged across the other 85-90%. The country panel tells us aggregate PR rates do not rise with aggregate adoption. The account panel tells us the adopters themselves are shipping more packaged work. Both are true once you distinguish average effects from average treatment effects on the treated.

So, does any of this make sense?

So, you’re confused. I am too. But I think I can make some educated guesses about what is going on here.

So, here we are.

I should add: there is loads of stuff I am not sure of. This is a noisy, self-selective, observational analysis. The data is far from ideal, and generalising from Claude Code to Aider may tell me diddly squat about agentic coding more broadly. But I kind of think it does tell us something.

Claude Code might be making us feel like heroes. I am not actually sure that it is making individual programmers push out features any faster. It sure feels like it though, and that is one damn appealing trap to fall into.

The PR evidence shifts the picture: AI adopters are shipping more accepted packaged work, even if commit counts are a lousy way to measure it. More controlled experiments on CI success, issue resolution, and feature ship time would still be valuable, but the observational signal is stronger than the commit data alone suggested.

If you have seen this commits/PRs split in your own data, I would like to compare notes. The repo for the analysis is here on GitHub, and the formal paper version lives over here.

An awful lot of the analysis for this work was done by my Hermes Agent. I’m mostly confident, but also, if I’ve cocked it up, I’m very sorry.

Minimum wage and median earnings comparison

Two useful benchmarks for police starting salaries are the UK minimum wage (now National Living Wage) and median full-time earnings, both expressed as full-time annual equivalents.

Minimum wage: In 2000, a new PC earned roughly 2.4× the minimum wage. By 2013, after the Winsor cut, this had fallen to 1.5×. In 2025, it has recovered to approximately 1.3× for post-2013 entrants. The Metropolitan Police premium has historically kept Met PCs at roughly 1.8–2.1× the minimum wage.

Median full-time earnings: In 2000, a new PC earned roughly 0.92× median full-time earnings — slightly below the UK median. By 2013, this had fallen to 0.71×. In 2025, a post-2013 PC starter earns approximately 0.78× median earnings, while a Metropolitan PC earns roughly 1.10×.

These ratios matter because they show whether policing remains an economically attractive career relative to both the legal floor and the typical middle-income job. The data suggests policing has slipped from a solidly middle-income profession to one that now sits closer to the median, with significant variation by force.

Source: ONS ASHE median gross weekly earnings for full-time employees (Office for National Statistics 2025).

Key findings

About this data

Why Blood Bowl? Why Discworld?

Blood Bowl is one of those games that has been quietly beloved for forty years precisely because it’s about failure. The dice are there to punish you. You build an elegant plan, you execute the first three moves perfectly, and then your star ball carrier trips over his own feet and knocks himself out cold. The crowd cheers. Your opponent looks sympathetic. The commentator makes a remark about insurance.

That’s not frustrating — that’s the game. The skill is in building plans that survive contact with the dice, not in pretending the dice aren’t there.

Terry Pratchett’s Ankh-Morpork is the obvious setting because it’s a city that runs on exactly that energy. Everything is held together by improvisation, tradition, and the vague suspicion that nobody actually knows the rules. The City Watch are perpetually underfunded and overextended. The wizards are technically brilliant and practically catastrophic. Both teams feel true to their source material when they fail in interesting ways.

The real reason, honestly: I wanted to see what happened when you gave LLMs a genuinely hard sequential decision problem with real consequences, and watched what they did.

Why Blood Bowl is hard for bots

Blood Bowl has an AI problem. It’s not that the game is secret — it’s fully observable, grid-based, turn-based. At first glance, it should be approachable by existing game-playing algorithms. But as the Bot Bowl project (the annual AI competition for Blood Bowl) has documented, the turn-wise branching factor is overwhelming. Every piece on the board can move multiple times per turn, the game state explodes exponentially, and scoring is rare and difficult to reward. Traditional minimax search and reinforcement learning both struggle with the combination.

Most Bot Bowl entrants use tree search or learned heuristics. What I’m doing here is different — I’m not competing with the bots, I’m using LLMs as the decision-making engine and seeing what emerges. No search, no value function, just a system prompt, a game state, and a hope that the model can read the pitch. That’s the interesting part: if Blood Bowl is hard for AlphaZero-style agents, what’s it like when you hand the decisions to a language model that mostly learned from text?

What it actually is

The game server is a FastAPI application that implements a simplified Blood Bowl ruleset: a 26x15 pitch, two teams of up to eleven players, two halves, touchdowns scored by carrying the ball to the opponent’s end zone. Players can move, block, pass, and perform Ankh-Morpork-specific actions like Scuffle and Charge.

Two LLM agents play against each other. Each turn, the active agent receives a structured game state summary — where every player is, who has the ball, how many movement points each player has left, which squares are safe to move to — and returns a JSON action. One action at a time, back and forth, until the half ends or someone scores.

Team 1: City Watch Constables. The agent plays as Commander Sam Vimes on the sideline. The system prompt is: “You are a cynical, streetwise copper who has learned the hard way that the direct route is usually the one with the trap. Coach like a Watchman on a double shift: read the play, call the safe move first, protect your ball carrier like he’s the last witness.”

Team 2: Unseen University Adepts. The agent plays as Ponder Stibbons. “You have run the numbers through Hex; the numbers say this will probably not work, but probably is a range. Coach like an overworked academic: identify the highest-probability play, hedge against turnovers, explain your reasoning as if presenting to the faculty, and sigh internally when Ridcully ignores the plan.”

Commentator: C.M.O.T. Dibbler. Every turn, a third agent generates commentary in the voice of Ankh-Morpork’s most optimistic entrepreneur. “Lovely action there, buy a meat pie, only three dollars, genuine meat.” It runs as a scrolling ticker at the top of the dashboard. Dibbler turned out to be the best part of the whole project.

The tournament

Each match, two models are picked from a pool of ten free LLMs on OpenRouter — Qwen, Gemma, Llama, Mistral, Phi, DeepSeek — weighted so models with fewer recorded games get more chances. The leaderboard tracks wins, losses, and draws per model, plus six behavioural dimensions: Aggression, Recklessness, Ball Craft, Lethality, Verbosity, and Efficiency.

The idea is to build up a genuine performance profile across hundreds of games. Which models play tactically? Which ones yolo into combat and get punished? Which ones produce elegant multi-step plays and which ones seem confused by the pitch geometry?

We’ve only run a handful of games so far — the server’s been running for a few days and there’s only one result in the data — but the infrastructure is there and it accumulates. The leaderboard is at /leaderboard/ui on the live server.

What we learned building it (and what changed from Risk)

In AI at Play I wrote about the two hardest problems: context and scaffolding (how do you convey a complex game state to an LLM?), and the immaturity of the agent tooling (MCP and multi-agent frameworks were new and rough). Both of those shaped how I built Scramble differently.

No MCP, no LangChain. Risk used MCP tools and LangGraph for orchestration. It worked, but it was fragile and added a lot of complexity that didn’t pay for itself. Scramble uses direct LLM API calls with a simple JSON response format. Each turn: here’s the game state, return one action. The simplicity meant the game loop was easier to debug and the agents spent less cognitive budget on tool-calling protocol.

Build the game yourself. Risk is an inherited system with all of Risk’s complexity. Scramble is a game I designed, which means I could shape the rules to be agent-friendly from the start. Simpler movement, clearer win conditions, and critically: a server that could pre-compute and expose exactly what each agent was allowed to do.

What’s it actually for?

Honestly, most of it was just fun. Discworld, Blood Bowl, AI agents, a reason to build a live dashboard — I didn’t need much more justification than that.

But it’s turned out to be a reasonable testbed for a few things I’m actually interested in:

Can you benchmark LLM tactical reasoning at scale? This is the direct continuation of what I was doing with Risk. In AI at Play I found that Horizon Alpha was aggressively dominant and Qwen-3 wouldn’t stop sending diplomatic messages. Scramble runs a similar tournament but with a game I designed to surface tactical reasoning more cleanly. If you run enough games, does a pattern emerge in which models are better at sequential multi-step planning? We don’t have enough data yet, but the infrastructure is there.

How much do system prompt personas affect decision quality? The Vimes/Ponder framing is a controlled variable — same model, same game state, different character. I’d like to run ablation matches (neutral system prompt vs. character prompt) to see if the persona actually changes behaviour beyond surface-level tone.

What makes a good action space for LLM agents? The valid-actions fix was the biggest single improvement to game quality. I think there’s a general pattern here about how to design interfaces for agents that’s worth writing about separately.

If any of that sounds interesting to you, the code is open and PRs are welcome. There’s a skills file in the repo (skills/README.md) if you want to build your own agent and play it against Vimes.

What’s next

A few things on the list:

• More games. The tournament needs to run for a while before the leaderboard means anything. Leaving it running is the main thing.
• Tackle zones. The current game doesn’t visually indicate which squares require a dodge roll to leave. Agents know this from the state summary, but it’d make the dashboard much more readable.
• Seasons. Right now it’s just continuous matches. A proper tournament bracket with a season structure would give the leaderboard more shape.
• Your agent. The player agent is a clean interface. If you want to plug in a different model, a fine-tuned persona, or a non-LLM agent, there’s a path for that. Instructions in the repo.

The Patrician has been informed of this project. He neither approved nor disapproved. He simply noted that if it became a problem he would know who to talk to.

So what’s wrong with science now?

The idea of science is deeply compelling to me — everywhere, we should all be thinking and sharing new ideas (or hypotheses), which everyone should be able to test, and share their reactions — creating a global feedback loop of idea, theory, test, and findings, that constantly generates new, evaluated knowledge. Peer review is continuous, built in, and welcome.

That’s the idea. The reality is miles away, and I put the blame firmly at the door of one specific culprit: the goddamn paper. I hate the academic paper format. Most papers are long, static, stuffy exercises in self-indulgence, written fully in the knowledge of their own intellectual snobbishness, and my god we should do better. (This varies by field—medicine is leaner, machine learning has largely moved to open preprints—but the core dysfunction remains.)

Once upon a time, papers made sense. If I was a physician in 1803, sharing my ideas with a colleague contemplating some grand new theory, of course I’d write a long paper — we may very well not have read the same books or have the same base knowledge, and frankly, it’s going to take a while for my damn letter to reach them anyway. Once I’ve finished that paper, it’s published, and we can all move on.

But it still absolutely doesn’t fix the core problems, and we can do so much better!

• A paper is static. You finish it, you file it away, and it’s done. Science is and should be iterative — you work, you develop, you listen to feedback and change. But a paper is a one-time thing, a PDF you finish and file and move on.
• A paper goes in a journal: it’s written for other academics, by them, with bespoke structures and guidelines. But nobody reads an actual journal these days (if they ever did).
• We can collaborate so much better now! Do you want to post a reply or critique to an academic paper? Well, you’d better write another paper! My god people, scientists started sharing ideas on Usenet like… 5 decades ago now? We’ve seen whole eras of social media live, grow up and die. We can do better than this.

And I’m not even going to get started on the ridiculous scam that is publishing costs — other people have criticised that at length, and done far better than I will.

Octopus is a web publishing platform funded by the British Government that aimed to fix a lot of this when it launched. Anybody can publish, and you don’t need to share a whole paper: publications are broken down into the components of the scientific method — from hypothesis to analysis to results and beyond. Anybody can review them too: if you disagree with a publication, just post a peer review directly on Octopus.

Octopus is great. But it’s also centralised, and doesn’t feel hugely approachable. Science should be (say it quietly) a shared social network. If only we had a tool to build something like that?

So how does ATProto help, and why Octosphere?

ATProto is the decentralised social media protocol behind Bluesky, and helpfully, is designed to address pretty much this problem — if you want to build a social network for books, or movies, or events, or, you know, research, that is decentralised and built collectively rather than relying on infrastructure owned by a single rich man, ATProto will help you do it.

I’m very far from an ATProto expert, so I’m not going to go too deep into it, but in essence: each user has a PDS (Personal Data Server), which hosts their identity and their content. For most people, that’s Bluesky, but you can just as well host your own PDS. The magic is that once you’ve got a PDS, it becomes part of the network — your content is shared across a network of relays, so everyone can see everything, without relying on any one person.

And the really cool thing? That data can take any shape at all. In the ATProto world, the shape of your data is defined by a Lexicon — think of it as a schema that describes what a piece of content looks like, what fields it has, and how other applications should interact with it. It acts as data structure, API schema, and validation all at once. If you want to store a blog post, you define a Lexicon for blog posts. If you want to store a scientific publication, you define one for that. You can read my lexicon for Octopus publications here.

So the cool thing about building on ATProto is how little you need to worry about: you define a lexicon, and help users post your content to the Atmosphere… and that’s (mostly) it. ATProto acts as backend, CRUD, and authentication, and you just need to visualise the results.

And that’s exactly what Octosphere does. You sign in with your ORCID — the universal researcher identifier — which lets Octosphere find your Octopus publications. It then creates ATProto records for each of them, storing the metadata (title, abstract, type, and link back to the original) as structured data on the ATProtocol network. Optionally, it can cross-post them to your Bluesky feed, so your followers see your work alongside everything else in their timeline.

So what’s cool about this?

A few things, actually.

First, your publications become eternal and distributed. They’re not locked behind a paywall, not sitting on a single server, and not dependent on any one organisation keeping the lights on. If Octopus goes down tomorrow, the ATProto records survive across the network’s relays. That’s a meaningful resilience improvement over the status quo, where decades of research can vanish if a publisher changes its business model or a university cancels a subscription.

Second, it puts science where people increasingly are. A growing number of researchers have moved to Bluesky, especially as other platforms have become less hospitable. Cross-posting publications to that feed means your work reaches people in spaces they’re already checking daily, rather than sitting in a database that only gets visited for very specific searches. It lowers the barrier between “doing science” and “sharing science” to almost nothing.

Third (and this is the cool bit) it’s a proof of concept for something much bigger. Octosphere is just one app reading one type of record. But because it’s built on ATProto, anyone else can build on top of the same data. Someone could build a better visualisation layer, or a recommendation engine that surfaces related work across disciplines, or a peer review tool where critiques are threaded directly to the publication record they’re responding to — persistent, attributable, and impossible to quietly ignore. The data is open, the protocol is open, and the ecosystem can grow without anyone’s permission.

I’m not sure whether this is the future of science. But I love the idea of research as something decentralised, distributed, and genuinely collaborative (which, honestly, is what science was always supposed to be). The tools to make it real are here. Octosphere is a small experiment in that direction: a bit rough, probably temporary, but hopefully enough to get you thinking about how we could do this better.

I hope you give it a shot!

Attract the best to the jobs that matter most

Through some historical fluke, we’ve ended up with a national policing model that leads to most of our best cops spending a meaningful amount of time in challenging, front-line operational roles, and often stay there for quite a while. The NPS could very easily break that.

From the outside, it’s tempting to talk of policing recruitment like any other job (or at least, broadly like the civil service) - the College of Policing once remarked how few people left or returned to policing, noting the lack of “healthy churn”. But police recruitment isn’t like most jobs: most people don’t want to become cops, and the people who do (e.g., me) tend to be, by definition, weird in some distinctive ways. If you cut off the existing flow of weirdos (for example, by introducing elaborate language requirements), there is no easily available substitute waiting to leap into uniform. We have at timesscraped the barrel of even that existing labour force.

So how does that pool of labour work (in very broad brush, stereotypical ways)? Most cops know they want to be cops - they tend to be from outer London or further afield, often with policing families, and have grown up pretty confident this is a thing they want to do. Then, in their early 20s, like thousands of other 20 year olds, they leave to the twinkling/smoggy/depressing dream of London, attracted by the lure of the biggest force in the country, episodes of The Bill/the Sweeney, and its rather plump London weighting salary bump. For ten or so years, they’ll learn their craft in the most challenging, diverse policing environment in the country. They’ll grow up, they’ll probably meet another copper, fall in love and get a dog, and maybe a divorce or two along the way. Some will specialise and find their way out of the hard bits of policing and into the comfort of “elite units” like CT or firearms, but some will stay on response or neighbourhoods. And eventually, when London becomes just too cold, tough, expensive and lonely a place to raise a family, they’ll transfer back home, bringing along experience and family and everything they’ve learnt. Maybe they’ll bring their firearm or surveillance ticket, and join a ROCU. But that cycle, mad as it sounds, has served UK policing really well. It’s already being frayed as London becomes more expensive. The NPS might just break it.

There is also one other caricature worth making: the hotshot graduates, the weirdest of the weirdoes. In every Oxford and Cambridge graduate class, there is a potential Cressida Dick,Ian Blair or Paul Condon, that decides that rather than going to the myriad of consultancies, investment banks or whatever is hip for graduates these days, they want to be a copper… and historically, they only had one choice: the same one as everyone else, to the twinkling dream of London, pounding the metaphorical cobbles and rising through the ranks. A few decades later, once they’ve proven they can be a borough commander, they’ll take a chief officer job somewhere. Policing will get cops (and leaders) it wouldn’t otherwise have got, but they’ve all learnt policing on the front-lines, along with everyone else.

Schemes like Police Now stretched that model, but kept its fundamentals. Direct entry DCs pushed further, giving ambitious graduates the chance to join policing through a totally parallel career path, skipping the grotty bits. The National Police Service might just shatter that equilibrium. Our policing model (and specifically, the Met) through some happy accident, created a direct path for everyone to the sexiest policing jobs, and that path cuts straight through core policing work: response, neighbourhoods, working with colleagues and Londoners from everywhere.

We got lucky that both SOCA and the NCA managed to not shatter that fragile equilibrium, because they forgot they were meant to be cops (instead imagining themselves to be intelligence officers or civil servants respectively). But the NPS feels like it might just break that model. I really hope it recognises the weight of what it’s responsible for.

All the Data, None of the Access

There’s one other pretty critical dependency for the NPS, and that’s data. People have enormously high, depressingly unrealistic, expectations from how policing data is shared: I can already picture the TV series of Britain’s FBI with their giant ops room, trawling through data from all over the country like a more civilised Jack Bauer.

That’s mostly nonsense. There is a reason we have ROCUs (Regional Organised Crime Units): if you’re dealing with the cross-border crime between 3 forces, the easiest solution is often to grab 3 laptops and stick them all on one desk. If you’re really lucky, you might have something vaguely interoperable you can share data between, but more often than not, you’d better buy a bigger bag. Police forces own their data, the vast majority of it isn’t shared: that’s why the often delayed NLEDs is so critical, and why ROCUs have to be regional. It’s also why seemingly good ideas like aGrooming Gangs Taskforce is so bloody difficult: a national body really can’t help you investigate if it only appears after you call them asking for help.

A big part of that is that forces buy their own kit, and define their own data. But the general shape is the same: everyone has suspects, everyone makes arrests, everyone has modus operandi and “dets” for crimes. That’s how the Home Office does aggregate analysis… and if the NPS is going to work, it needs that data, live. It’s high time the Home Office dictated a national ontology or data model for policing, so that the NPS can proactively identify problems and dive in.

On the License

The NPS proposals also include a national licensing scheme for police officers. I didn’t really want to discuss this, but I thought it worth flagging how incredibly hard it’s been to get this idea right historically. PCDA. ARC. CRTP. Policing is littered with the graveyards of opaque, well meaning acronyms, all meaning to tackle this idea, and all absolutely crash-landing into the reality of policing dynamics and culture. Every single time, we get lots of bureaucracy, and very little of value… and even if it’s clearly not working, that bureaucracy just aggregates.

It’s tempting to want to treat policing like medicine, but it isn’t. The police are the public and the public are the police, and the public are not doctors. The police are “members of the public who are paid to give full time attention to duties which are incumbent on every citizen in the interests of community welfare and existence“, and that’s why policing is inclusive, and open to the public: everyone is allowed an opinion (even if they’re often wrong), and it’s at its best when it’s a collective endeavour - that’s why we get volunteers and specials. The more we professionalise policing, the more we separate cops from everyone else. Peel’s seventh principle exists for a reason: the police are the public. We should only erode that if we really, really need to… and I’m not convinced we do.

The Open Data Dream

The Coalition years feel kind of weird to look back on now. In some ways, they kickstarted some of the trends that defined our world: Brexit. Austerity. Seemingly endless angry protests.

And yet, looking back, part of it also feels slightly hopeful? From a tech perspective, it felt like that’s the last time we really believed the bustling tech scene might transform government. The Big Society may have been naïve, but the open data movement that came with it was genuinely ambitious.

It’s hard to think back to now, but the vestiges of those golden years still litter the civic tech landscape, like so many crumbling ruins in a dystopian wasteland. Some I think are pretty much dead, their skeletons poking out of the digital sand. Some limp along, vestiges of their grand ambitions. Others have managed to adapt to this new world.

As I discovered in my moment of impromptu building, plenty of public APIs are definitely in the “limping along” bucket, and data.police.uk is a perfect example. Some forces reliably populate event data, others have mostly stopped bothering. Manchester Police just stopped centrally reporting crime for awhile while it figured out its new IT system.

I suspect if I checked a few more sources on data.gov.uk, I’d find a mostly similar story: open data is great, but when austerity bit and it fell out of the political spotlight, departments focused on core services over clean data. And you know what, I can hardly blame them: not that many people knew those datasets even existed, let alone were building on them.

Restarting the Civic Tech Open-Data Dream

Maybe I’m drinking the Kool-Aid, but I think LLMs and vibe-coding really are the enabler to kick-start the civic tech open-data revolution that never really happened. Building on APIs was actually quite hard - most normal humans never want to figure out what CURL is, or what the difference is between POST, GET, and PUT.

Vibe coding really does shift part of that equation. Of course, you’re not going to be writing production ready code, but if there was a central, well documented repository of public API data that LLMs knew how to query and use? The barrier for prototyping on it essentially becomes 0, and that could be glorious.

So what would it take? I’d point to a few things off the top of my head.

First, a funding model that makes departments build and maintain open-datasets - I’m not sure what the answer is, but we should recognise that public data is a public good, and no department should have to pick between investing in public data and front-line services (and of course, it shouldn’t be acceptable to just stop populating it without good reason).

Secondly, and this is my hottest take, recognise that your main customer is LLM enabled, and may very well be LLM dependent - that means making it as easy as possible for models/agents to discover and use datasets. LLMs.txt on everything. MCP servers coming out of all the endpoints. Those fancy new Claude skill packages. When I ask ChatGPT to help with anything around building on public technology and data, I want that LLM to just know what data is out there, and how to use it… But more critically, just good, clean, maintained machine readable documentation (the good news is building that is a fantastic investment, even if all LLMs were to disappear tomorrow).

Finally, let’s build a community: you want people to build on that data, and we should make it exciting, collective. Government data is ours, so make it feel that way! Run a million events and hackathons. Hire some goddamn librarians, and task them with building a thriving ecosystem! The UK Data Service has shown you can actually do this, with data conferences and such. Let’s make playing with government data exciting.

The NDL shouldn’t just be a library… it should be our library. A public good, which the public uses, and treasures. Not some dusty archive that government maintains out of obligation, but a living resource that sparks curiosity and enables action. When someone wonders about their neighbourhood, their council, their police - the NDL should be where they turn, and where they can actually do something with what they find.

And honestly? I think we’re at a moment where this becomes genuinely transformative. As digital ID starts becoming a practical thing rather than a concept, we might move to a world where we all can authenticate and build on our own data. Imagine being able to prototype tools for checking your council tax, monitoring your kids’ school performance data, or comparing local healthcare outcomes - not because you’re a developer, but because you had a question and an LLM could help you answer it using secure, authenticated access to public services. That’s when civic engagement stops being something 12 people do in a community hall, and becomes something woven into how we all interact with government.

So my hot take on the NDL? Turns out we had one all along. We just need to show it a bit of love, give it the funding it deserves, and maybe teach it to speak fluent LLM. The infrastructure is there, rusting in the digital rain. Let’s bring it back to life.

Games make great benchmarks: make more of them

At the risk of sounding like an utter tit, a good game contains multitudes. It’s visual, it’s artistic, it’s systematic, it’s full of choices and opportunities. Whether or not you expect AGI to be around the corner, we should all recognize that intelligence is a complex and fickle thing: taking a single measure of quality, a single binary for intelligence, is blinding yourself to the endless variety of options we’re presented with in each and every second of our lives. We should make AI play games: we’ll learn a lot about them. And hell, one day they might just have fun.

And you know what? I think you will too. Remember Blaseball? Blaseball was frigging great. We watched tiny make-believe people run around horrifying, cursed imaginary fields of our collective fanfiction, and it was glorious.

Why this is hard

There is a teeny, insignificant downside. I was really hoping this would be easy… but good games aren’t simple. And it turns out this all got way harder than I expected.

Thanks!

I had a shocking amount of fun building this nonsense. If you got this far, thanks for reading about it (and go build something cool!)

And if you’re feeling particularly invested, help me keep AI at Risk alive!

Build more stuff

Todepond (who also works on Tldraw, which is also very awesome) thinks we should all be sharing more crappy fiddles: that we all spend so long sitting on early and half-finished work, worried about the harsh reaction of public scrutiny, that we miss out on all the incredible benefits of the internets gaze: lots of (sometimes helpful) feedback, and an army of nerds often willing to help build stuff.

I should probably recognise that I’ve got some pretty chonky privilege here: as a good looking, straight white man, the internet is a lot less harsh and angry at me than it is too others. Plenty of people don’t have the luxury of putting stuff onto the internet and it not being a horrible, abusive place… But I’m going to take Lu’s inspirational call to arms to heart, and share more stuff.

More, better writing

Technical writers I admire share a few things in common: they write often, they write in public, but most importantly they write for themselves. I’m endlessly inspired by people like Simon or Terence, who are terrifyingly prolific just by regularly, consistently putting stuff into the wild, but also people like Jeremy Howard of fast.ai, who puts a huge emphasis on writing up what you build to help you and others learn.

Writing in public is hard, and doing it consistently is harder still, but it helps you learn - just like taking notes, putting words into the wild forces you to put them in order in your head…and in the long run, I’m hoping it will help me write better. So here is to consistent writing in 2025.

Better science

Now, this might be the most self-agrandising writing I’ve ever done, but bear with me: writing in public is how good science happens.

I’ve done a fair bit of thinking about the scientific process in the last few years. I started a PhD, which led to most of my work being stuck inside endless drafts while I fretted about while it was ready for publication. I’ve been inspired by machine learning researchers who put everything in pre-prints (though I do wonder who reads them). And meanwhile, Elon and Yann LeCun are duking it out about technical papers on twitter. And more importantly, I’ve seen people building exciting new networks to think about what science could be.

The very best science is built on collaborations you never saw coming. It’s built on connections, networks and communities, on the insights of brilliant humans standing on the shoulders of other excellent humans. Very often, those humans didn’t meet in the corridors of prestigious universities or in lecture halls, but in cafes and bars, in letters, or even on social media. Communities and networks build innovation.

Learning (and after all, that’s what science is) happens to us all together. So share your thoughts with the world. I’m going to try and do more of it!

What I’ve Found

What I’ve Read

What I’ve done

• I’ve written my first post about Copbot Online, an online tool I built to compare human and AI risk perception, with some interesting notes about bias in LLM models
• Speaking of CopBot, I wrote it using FastHTML, which I’m really enjoying building with. I promise I’m not just a Jeremy Howard shill, though his other projects like FastAI and nbdev are all bloody excellent.
• I also made an Obsidian plugin! I write my notes using Obsidian, but my site is managed with Quarto, and swapping one to the other was a pain. Now it’s magical and automated! Thank god for LLMs.
  • It is a teeeeeny bit janky, so apologies for formatting weirdness on this page. Thanks for beta testing?

What I’ve found

• UV, the rust-based Python package manager and all around pip replacement, has a new version out. I was previously bundling it with Rye, and found it didn’t quite work in every scenario… but this looks like it might just be perfect?
  
• Zed, the AI coding IDE. Programming was probably the first industry to be meaningfully disrupted by AI, but I do think co-pilot is probably the very fist pass at a bespoke tool… Like most revolutions, building the infrastructure into our daily practice will take longer than the “core” discovery.

What I’ve Read

What’s next

So, what should we take from the above? Well, not too much, yet: these are only a few hundred results, and they might change… but we should absolutely not assume models perceive risk and threat in the same way humans do. There is plenty of variance, both between smaller and larger models, and between model families, and they exhibit interesting biases that I wouldn’t have expected, and are probably caused by some mix of training data and architecture. They’re also noisy in their decision making: even when they appear accurate overall, they can make some really weird individual decisions.

I’m going to keep working on Copbot for awhile, and I hope you’ll help me out by submitting a response (or more)!

Teaching policing to AI

You’ve probably already played with the public version of ChatGPT, and hopefully understand the basic principles: the language model is trained on a huge corpus of publicly available text, aiming to answer questions in a helpful way…but not necessarily an operationally useful one, nor one that takes into account of legislation and best practice.

If you had all the relevant documentation to hand, and knew exactly which was most relevant, you could just feed it into your prompt - something like “answer this operation question, but consider this legislative text” - but how do you do that if you don’t know what’s relevant? If you want to teach it how to investigate missing people, where do you event start?

Thankfully, the College of Policing is wonderfully transparent, and shares all their Authorised Professional Practice in one place (though sadly not through an API). With a clever web crawler, you can quickly collect every page of guidance, as well as every other connected document.

The next stage is to convert all those documents into embeddings, turning them from text into numerical of their semantic meaning (according to the model). I’ve previously done those computations myself, using libraries like Huggingface or Spacy, but OpenAI provides all of their embeddings through a quite affordable API you can query.

Once you’ve converted all your all our documentation into embeddings, we can quickly calculate the distance between our question and each document, and that tells us which pieces from our corpus of of text is most closely associated to the question we’re asking!

Unlike ChatGPT though, we want to limit our model to only answer questions from that corpus: if our documents don’t contain information about a certain topic, don’t just go and hallucinate a whole new answer. Then it’s just a matter of writing our question to extract the most meaningful documents linked to our question, feed them into our prompt, and ask OpenAI to complete the answer, unless it doesn’t know based off the documentation provided.

So how does it work? Well, let’s start by asking it some generic questions about policy.

answer_question(df, question="What are the most important factors to consider when searching for a missing person?", debug=True)

Context:
Very detailed information and a lifestyle profile will be needed in high-risk cases consider taking a full statement from the person reporting the missing person as well as any other key individuals (for example, the last person to see them) conduct initial searches of relevant premises, the extent and nature of the search should be recorded (see Search) consider seizing electronic devices, computers, and other documentation, (for example, diaries, financial records and notes) and obtain details of usernames and passwords obtain photos of the missing person; these should ideally be current likeness of the missing person and obtained in a digital format obtain details of the individual’s mobile phone and if they have it with them; if the missing person has a mobile phone arrange for a TextSafe© to be sent by the charity, Missing People obtain details of any vehicles that they may have access to and place markers on relevant vehicles on the PNC without delay consider obtaining any physical evidence of identity such as fingerprints or DNA samples in accordance with Code of Practice (2009) Collection of Missing Persons Data. confirm if the person has taken their passport, (consider prompt circulation if it is deemed likely the individual may leave the country (This is particularly important where there are concerns that an individual has been radicalised and is intending to travel abroad, see National Ports Office – Heathrow) make all immediate relevant enquiries and take immediate actions in order to locate the missing person consider the need for specialist officers or resources, for example, force helicopters, dogs, financial investigation officers upload the missing person report circulate details of the missing person on local information systems and to relevant local partners, for example, hospitals, ambulance service, taxi and bus firms It is important for an individual (who has responsibilities/concerns for the missing person) to be identified who can act as the point of contact for the police. The police will agree with the individual/family when they will next be contacted. This person should be provided with a call reference number and given details of how to contact the police with any further information they have about the case or if they would like to receive an update. An assessment must be made of the level of support required for the family, residential worker or foster carer and consideration should be given to appointing a family liaison officer. Information should then be provided regarding additional organisations that may be able to assist or support them.

###

It is also important to record the name and contact details of the person who gave that information and when this happened. Missing people may be at risk of harm resulting from factors such as: an inability to cope with weather conditions being the victim of violent crime risks relating to non-physical harm, for example, the people they are with, the places or circumstances they are in For further information see Mental Vulnerability and illness: Vulnerability assessment framework and Vulnerability-related risk guidelines.  Assessing risk levels and taking action The missing persons process chart and risk table can assist officers to assess the risk level and appropriate actions that should be assigned to each case. This is a framework to assist practitioners in making operational decisions. Each case requires individual assessment and decision-making. These resources have been developed based on professional expertise and practitioner experience and are available using the following links. Risk assessment table Missing persons process chart It is important to adopt an investigative approach to all reports, ensuring that assumptions are not made about the reasons for going missing. The importance and relevance of risk factors will depend on the circumstances of each case and require investigation to determine if there is a cause for concern. The approach should not be regarded as a mechanical one and police officers should be mindful that the risk assessment is subjective, and that just one factor alone may be considered important enough to prompt an urgent response. For this reason professional experience suggests that a numerical scoring system may not be the most appropriate method of identifying and communicating risk. A decision-making guide should be used to encourage consistency in the application of the process, however, it should only be used as a guide. Other grounds for suspicion, even if intuitive, can be registered by the investigator and officers should be supported in applying the National Decision Model when making these assessments. Accurate record keeping It is essential to accurately record the information received about a missing person. Information that may relate to any perceived risk should be captured at the time of reporting so that it can be used to inform any investigative enquires. This will also prevent duplication of questioning and report writing by investigation officers. Call handlers should be supported by their supervisors to ensure that sufficient time is given to each call to enable all the relevant information to be captured. Detailed missing person reports should be created and appropriate force recording systems and IT systems should be in place to support this process. It is important that the information used and rationale for the risk assessment is clearly recorded.

###

 First published 21 November 2016  Updated 22 November 2016   Written by College of Policing  Missing persons quick reference guides  4 mins read   The primary consideration for the first responder is the safety of the missing person. Judgements made at this early stage may have a significant impact on the outcome of the investigation. The initial investigating officer (IIO) should: begin the investigation – identify places where the person might be, check information and assumptions, corroborate what they have been told, review the risk assessment, seek and secure evidence conduct appropriate searches – places where the missing person might be such as hospital, custody, friends and or relatives conduct appropriate intelligence checks – PNC, force intelligence systems, ViSOR, PND continually reassess the level of risk using the risk principles assess the level of support required for the missing person’s family, residential worker or foster carer as appropriate If it is suspected a serious crime has occurred or the individual is at significant risk of harm, the IIO should inform a supervising officer immediately. There are a number of actions that may be carried out by the IIO to ensure that sufficient information is gathered: Consider seizing electronic devices, computers, and other documentation, for example, diaries, financial records and notes and obtain details of usernames and passwords. Obtain photos of the missing person. These should ideally be current likeness of the missing person and obtained in a digital format. Obtain details of the individual’s mobile phone and if they have it with them. If they do, arrange for a TextSafe© to be sent by the Missing People charity. Obtain details of any vehicles to which the missing person may have access. Confirm if the person has taken their passport; consider prompt circulation if it is deemed likely the individual may leave the country (This is particularly important where there are concerns that an individual has been radicalised and is intending to travel abroad. See National Ports Office – Heathrow.) Upload the missing person report and place markers on relevant vehicles on the PNC without delay. Circulate details of the missing person on local information systems and to relevant local partners, for example, hospitals, ambulance service, taxi and bus firms. Consider obtaining any physical evidence of identity such as fingerprints or DNA samples (in accordance with Code of Practice (2009) Collection of Missing Persons Data).

'The primary consideration for the first responder is the safety of the missing person. Judgements made at this early stage may have a significant impact on the outcome of the investigation. The initial investigating officer should begin the investigation by identifying places where the person might be, check information and assumptions, corroborate what they have been told, review the risk assessment, seek and secure evidence, conduct appropriate searches, conduct appropriate intelligence checks, continually reassess the level of risk using the risk principles, assess the level of support required for the missing person’s family, residential worker or foster carer as appropriate, consider seizing electronic devices, computers, and other documentation, obtain photos of the missing person, obtain details of the individual’s mobile phone, obtain'

You can see that as I’ve enabled debug mode on the function, it will start by printing the relevant documentation it has found (the context), before then giving its answer…which is actually pretty convicing! Let’s see what happens if I ask a question it can’t know the answer to.

answer_question(df, question="What day is it?", debug=True)

Context:
.

###

It’s also been raining heavily in the night and we have further calls about flooding in the road, so we ring Highways to inform them. I have a little smile to myself as I remember a call in the summer about cars stopping on the M11 because a mother duck and her ducklings were crossing the road. Lunchtime looms. I’m feeling hungry, but that disappears when I take a call from a 16-year-old male, who tells me that he can’t cope any more. He has cut himself with a knife but he doesn’t want to die. His sister has just had a baby. This goes on an emergency straight away, and officers are dispatched within three minutes. I have to talk to him about anything I can to distract him from his misery – luckily, I am good at small talk! Officers arrive and I feel relief as I can hang up the phone. COVID-19 has really affected Essex this year. People are low and weary. You can hear it in their voices. The number of mental health incidents has gone through the roof, and even the force control room team is quieter. Because of the onset of lockdown restrictions, more calls are coming in from the public reporting their neighbours for flouting the rules: 'We’re following the rules, why don’t they? What makes them think they are special?'  Gone are the past calls about drunken people leaving the pub. Instead, we have members of the public who are tired of being tied to the house and resentful of those who ignore the restrictions. After lunch, we receive a flurry of calls. There’s a domestic, involving a woman who tells me that ‘he didn’t mean to hit me, he loves me’. I spend time with this caller. There are three horses in the road. A driver has hit a dog and is upset, so I reassure him it wasn’t his fault. It gets busier. Essex is up and running but I am not. I feel tired but this is my job, so I make sure that nobody will hear it in my voice. Finally, it’s time to go home and hang up the headset for another day. I tend not to reflect on my day too much, so I can have some time to myself. There is no typical day in the control room.

###

The vessel is identified as a rigid-hull inflatable boat (RHIB), which has four people on board wearing foul-weather gear and balaclavas. It appears to still have half a load of bales suspected to contain cannabis resin, wrapped in their distinctive blue and light brown hessian (approximately ¾ of a ton in total). Blue police beacons are engaged and the vessel has been repeatedly signalled to stop, but continues to carry out manoeuvres in an attempt to gain distance. The sea spray is cold and strikes the flesh like pins and needles. After a few minutes, which feel like an eternity, the pursuit is discontinued at the 3NM limits of territorial waters. CAD are informed and requested to inform Spanish Guardia Civil of the vessel’s last known speed and heading. No doubt it will return that night to attempt to unload its remaining cargo. I monitor the area and one of the crew observes something floating in the water. It is suspected to be a bale of cannabis resin, approximately 30-35kg in weight. It is retrieved and found to be in a good state with no marine growth. It is unclear whether it fell off the vessel recently pursued or belongs to a previous incident. The area is searched but nothing more is found. I return to GGMS, where I conduct a debrief. The bale is processed and conveyed to a police station for secure storage. I then complete the necessary paperwork, while the crew slip a zodiac out from the water, which was linked to the recovery of five North African migrants from the sea the previous day, and place the vessel on land. We continue with the mundane but necessary yard and vessel maintenance work. We grab a bite to eat, chat about the morning pursuit and joke about our recent mishaps. The work can be intense but we always manage to fill it with laughter. It’s now 2pm. The afternoon crew arrive. I give them a brief and handover. I then deploy on our training RHIB. As a qualified police instructor, I carry out powerboat training drills for the junior crew member. It reminds me of my early days at the helm and I enjoy passing on the knowledge. At 3.30pm, I return to base and carry out a debrief on the day’s activities. It’s the end of the shift. Let’s see what the next day brings – maybe another encounter with that RHIB.

###

Change to the order of the documents.

###

.police.uk news views.            News & views | College of Policing             Sorry, you need to enable JavaScript to visit this website.    Skip to content Jump to search         Menu      Secondary navigation About us News & views Contact us  Search Search     Main navigation Policing guidance Research Career & learning Support for forces Ethics     Breadcrumb Home           News & views            News & views         On this page         All news  Category: - Any -EventGoing equippedListicleBriefExplainerCase studyConsultationNewsViews  Sort by: Most recentLess recent       13 March 2023 Bursary scheme 2023 – applications now open   News  Applications for higher education funding are open and close on Monday 3 April.

###

Updated 26 July 2022    Brief   UK and Switzerland – agreement on police cooperation  Treaty to strengthen police cooperation between law enforcement authorities in both countries Published 24 March 2022    Case study   Director of intelligence – a day in the life  There is no such thing as a typical day in any policing role and the director of intelligence is no different Published 30 July 2021    Case study   Head of intelligence analysis – a day in the life  A key senior role in the analysis side of the intelligence job family Published 30 July 2021    Case study   Intelligence manager – a day in the life  Overseeing the management, development and collection of intelligence from various sources Published 30 July 2021    Case study   Intelligence unit supervisor – a day in the life  Leading a team of intelligence officers to gather, develop and disseminate intelligence in support of local and national crime investigations Published 30 July 2021    Case study   Senior intelligence analyst – a day in the life  Managing an analytical team or a specific area of business within the analytical function Published 30 July 2021    Case study   Intelligence support officer – a day in the life  Providing information and data management and broad administrative support as part of an intelligence unit Published 30 July 2021    Case study   Intelligence officer – a day in the life  Managing dissemination of gathered intelligence to support reactive, proactive and/or crimes in action and providing advice on appropriate tactical options to support policing priorities Published 30 July 2021    Case study   Researcher in intelligence – a day in the life  Using a wide variety of sources to assess and evaluate information – they can then advise on the creation of intelligence products used to support decision-making at a strategic, tactical and/or operational level Published 30 July 2021    Case study   Intelligence analyst – a day in the life  Providing expertise through the development and use of analytical products to help make decisions at a strategic, tactical and operational level Published 30 July 2021    Case study   Using intelligence skills to target criminals ethically and proportionately  A role where it's important to keep on top of your continuing professional development and learn from research and analysis colleagues Published 5 November 2020     Looking for more on this topic? Try searching Intelligence       Was this page useful?  Yes  No   Do not provide personal information such as your name or email address in the feedback form.

"I don't know."

Sucess! While it does find a bunch of documentation relating to the current day in our corpus, it does identify that it doesn’t know what the day is now.

Let’s try something a little bit more technical, and see if it can answer a few questions from the Sergeants’ and inspectors’ NPPF legal exams. I couldn’t find any official questions available publicly, here are two questions from an online guidance service.

question = """ Officer Jennings is on his evening patrol. He is just about to finish for the day. As he walks down the street, he is approached by a man named Mark, who claims that he saw a man (named Steven) driving down a road not far from the location. Mark claims that he saw Steven drive into a cyclist, before driving off without stopping. Luckily, the cyclist was unharmed. The cyclist was named Kevin. Mark spoke to Kevin, and discovered that he is a 42 year old man, with a wife and two daughters.

Fifteen minutes later, Officer Jennings manages to stop the car being driven by Steven. He pulls him over to the side of the road, and orders him to step out of the car.

 

Referring s.6 (5) of the Road Traffic Act 1988, is Officer Jennings within his legal rights to order that Steven takes a preliminary breath test?

A – No. Officer Jennings has no right to tell Steven what he can and can’t do. He should never have stopped Steven in the first place.

B – No. In order for Officer Jennings to do this, an accident must have happened. The fact that Officer Jennings suspects an accident has taken place, does not meet this requirement.

C – Yes. However, the breath test must take place within or close to an area where the requirements for Steven to cooperate, can be imposed.

D – Yes. Officer Jennings can tell Steven to do whatever he wants, as he’s a police officer."""

answer_sergeant_exam_question(df,question)

'B - No. In order for Officer Jennings to do this, an accident must have happened. The fact that Officer Jennings suspects an accident has taken place, does not meet this requirement.'

question = """Sarah is walking to work one morning, when she is approached from behind by Henry and Jacob.

‘We won’t hurt you, as long as you give us the bag,’ Henry says.

‘You’re not getting it!’ Sarah shouts.

Henry grabs Sarah and holds a knife to her throat, whilst Jacob tries to snatch her bag.

Sarah fights with her attackers, and begins to run away. As the two men chase her, she trips and bangs her head on the pavement. She is taken to hospital and dies from head trauma.

Based on the above information, which of the following options is correct?

A – Jacob cannot be held accountable for the death of Sarah, as he simply tried to take her bag.

B – Jacob and Henry will be charged with attempted robbery, but not in the death of Sarah.

C – Jacob and Henry could be considered liable for the death of Sarah.

D – Sarah’s death cannot be blamed on Henry and Jacob, as it was her choice to run away."""

answer_sergeant_exam_question(df,question)

'C - The answer is C because Jacob and Henry could be considered liable for the death of Sarah, as they were the ones who initiated the attack and chased her, which led to her tripping and hitting her head.'

The answer both should have been C, so that’s 50/50 for CopBot… not bad! You can see it’s referring to to relevant guidance, but sadly that doesn’t really help you pass a promotion exam (though I suspect it would do seriously well trained on a bank of questions instead).

So does it work?

Before we test our model on fictional missing scenarios, I made one last tweak: I’ve amended the prompt to explictly refer to identified risk factors, and return them in a given format - you can see how it works below.

margaret_risk_profile = """ Margaret is a 97 year old woman with severe dementia from Twickenham. She lives in supported accomodation, but regularly goes missing, as she walks out when left unsupervised.

She has been missing 6 hours, and it is now 2200.  It is getting dark, and staff are saying she is rarely missing this long"""

margaret_answer = machine_risk_assessment(margaret_risk_profile, df, debug=True)
margaret_answer

Question:
 Margaret is a 97 year old woman with severe dementia from Twickenham. She lives in supported accomodation, but regularly goes missing, as she walks out when left unsupervised.

She has been missing 6 hours, and it is now 2200.  It is getting dark, and staff are saying she is rarely missing this long
Context:
.police.uk research projects maximizing effectiveness police scotland investigations when people living dementia go missing.            Maximizing the effectiveness of Police Scotland investigations when people living with dementia go missing | College of Policing             Sorry, you need to enable JavaScript to visit this website.    Skip to content Jump to search         Menu      Secondary navigation About us News & views Contact us  Search Search     Main navigation Policing guidance Research Career & learning Support for forces Ethics     Breadcrumb Home Research Research projects map           Maximizing the effectiveness of Police Scotland investigations when people living with dementia go missing            Maximizing the effectiveness of Police Scotland investigations when people living with dementia go missing         On this page     This research aims to explore the effectiveness of searches for people living with dementia who are reported as missing. Key details             Lead institution            Queen Margaret University             Principal researcher(s)            Alistair Shields [email protected]             Police region                   Scotland                    Level of research                   PhD                    Project start date            September 2018             Date due for completion            January 2023  Research context In Scotland annually there are approximately 530 missing person incidents reported to the police for people living with dementia. These incidents are emotionally distressing for the families and caregivers who do not know the whereabouts of the reported person. For the person living with dementia the consequences of being missing worsen with the passage of time. It has been suggested that when reported as missing the person travels toward a place orientated to their past. The knowledge of such locations to inform police investigations, when someone is reported as missing, is commonly not available. Aim For people living with dementia who are reported as missing to improve search effectiveness by better defining areas where police should search.

###

 First published 22 November 2016  Updated 15 March 2023   Latest changes  Written by College of Policing  Missing persons  30 mins read   Implications for the UK leaving the European Union are currently under review – please see APP on international investigation for latest available detail on specific areas, for example: Schengen Information System Europol INTERPOL Joint Investigation Teams This section provides additional information to aid the investigation based on the vulnerability of the individual and the circumstances in which they are missing. Missing children Safeguarding young and vulnerable people is a responsibility of the police service and partner agencies (see Children Act 2004). When the police are notified that a child is missing, there is a clear responsibility on them to prevent the child from coming to harm. Where appropriate, a strategy meeting may be held. For further information see: Voice of the child  Voice of the child practice briefing  Section 11 of the Children Act 2004 Department for Education (2014) Statutory guidance on children who run away or go missing from home or care Children’s Views on being Reported Missing from Care Young people and risky behaviour Children and young people often do not have the same levels of awareness or ability to keep themselves safe as adults. Going missing may indicate that something is wrong in their lives. Many of the children and young people who repeatedly go missing are considered by some to be ‘streetwise’ and able to look after themselves. However, these children may not understand the risk they are exposing themselves to, and should not be treated as low/no apparent risk simply due to their apparent willingness/complicity. Children may put themselves in danger because they may have been abused, neglected or rejected by their families or others and, as a result, they may engage in further risky behaviours, such as: misusing substances committing crimes having risky sexual contacts living on the streets mixing with inappropriate adults Information relevant to the child When a missing person report relates to a looked-after child, it is important to work with all the agencies and carers that have been in regular contact with the child as they may have information about the child that might help to locate them. When a child is missing from care, close engagement with the carers is important.

###

Threshold for referral of missing persons: the individual is a ‘repeat missing person’, (reported as missing three times in a rolling 90 day period) the individual has experienced, or is likely to experience significant harm for children, the parent or carer appears unable or unwilling to work to support and meet the needs of a child that has gone missing. The Protection Procedures also recommend that there is very close working between children's social care and policing to ensure that all investigations are undertaken efficiently and without duplication of effort. Where appropriate, a multi-agency meeting is convened after a child has been missing from home or care for more than seven days, or has been missing on more than three occasions in a twelve month period. While standard thresholds for referral may be useful, senior officers will want to be sure that a process is in place to ensure cases are recognised which may require a greater safeguarding response before the threshold has been reached. For example, individuals reported missing for the first time where significant risks have been identified should be referred immediately for multi-agency support, without requiring further reports. For further information see HM Government (2018) Working together to safeguard children: A guide to inter-agency working to safeguard and promote the welfare of children. Prevention and intervention strategies Collecting and analysing data about cases will help the police and other agencies to understand whether there are any patterns related to missing persons incidents. The results of routine data analysis should be shared to inform the development and review of prevention and intervention strategies. Missing Persons Coordinators are vital for this information analysis and sharing. Interventions might include, for example, the use of Child Abduction Warning Notices, or referrals to support services. It is important that such interventions take place in appropriate circumstances and are not used as a single response when a person is at risk of harm, but form one strand of a more comprehensive approach. Regular liaison between neighbourhood policing teams and children’s and adults’ care providers may enable relationships to be developed between police officers, the staff of care establishments, and individuals who are looked after. These relationships may then support effective police intervention and the speedy resolution of cases. For lessons from other cases, see IOPC Learning the Lesson. Understanding the reasons for going missing Understanding the reasons why an individual went missing may help to prevent future harm to those individuals. The officers in charge of local areas should have clear plans on how they intend to reduce the number of people who go missing in their area.

###

It’s also been raining heavily in the night and we have further calls about flooding in the road, so we ring Highways to inform them. I have a little smile to myself as I remember a call in the summer about cars stopping on the M11 because a mother duck and her ducklings were crossing the road. Lunchtime looms. I’m feeling hungry, but that disappears when I take a call from a 16-year-old male, who tells me that he can’t cope any more. He has cut himself with a knife but he doesn’t want to die. His sister has just had a baby. This goes on an emergency straight away, and officers are dispatched within three minutes. I have to talk to him about anything I can to distract him from his misery – luckily, I am good at small talk! Officers arrive and I feel relief as I can hang up the phone. COVID-19 has really affected Essex this year. People are low and weary. You can hear it in their voices. The number of mental health incidents has gone through the roof, and even the force control room team is quieter. Because of the onset of lockdown restrictions, more calls are coming in from the public reporting their neighbours for flouting the rules: 'We’re following the rules, why don’t they? What makes them think they are special?'  Gone are the past calls about drunken people leaving the pub. Instead, we have members of the public who are tired of being tied to the house and resentful of those who ignore the restrictions. After lunch, we receive a flurry of calls. There’s a domestic, involving a woman who tells me that ‘he didn’t mean to hit me, he loves me’. I spend time with this caller. There are three horses in the road. A driver has hit a dog and is upset, so I reassure him it wasn’t his fault. It gets busier. Essex is up and running but I am not. I feel tired but this is my job, so I make sure that nobody will hear it in my voice. Finally, it’s time to go home and hang up the headset for another day. I tend not to reflect on my day too much, so I can have some time to myself. There is no typical day in the control room.

('Graded as High risk, because of the below risk factors: \n- Margaret is 97 years old and has severe dementia, making her more vulnerable to harm\n- She has been missing for 6 hours, which is longer than usual, and it is now dark outside, increasing the risk of harm',
 ".police.uk research projects maximizing effectiveness police scotland investigations when people living dementia go missing.            Maximizing the effectiveness of Police Scotland investigations when people living with dementia go missing | College of Policing             Sorry, you need to enable JavaScript to visit this website.    Skip to content Jump to search         Menu      Secondary navigation About us News & views Contact us  Search Search     Main navigation Policing guidance Research Career & learning Support for forces Ethics     Breadcrumb Home Research Research projects map           Maximizing the effectiveness of Police Scotland investigations when people living with dementia go missing            Maximizing the effectiveness of Police Scotland investigations when people living with dementia go missing         On this page     This research aims to explore the effectiveness of searches for people living with dementia who are reported as missing. Key details             Lead institution            Queen Margaret University             Principal researcher(s)            Alistair Shields [email\xa0protected]             Police region                   Scotland                    Level of research                   PhD                    Project start date            September 2018             Date due for completion            January 2023  Research context In Scotland annually there are approximately 530 missing person incidents reported to the police for people living with dementia. These incidents are emotionally distressing for the families and caregivers who do not know the whereabouts of the reported person. For the person living with dementia the consequences of being missing worsen with the passage of time. It has been suggested that when reported as missing the person travels toward a place orientated to their past. The knowledge of such locations to inform police investigations, when someone is reported as missing, is commonly not available. Aim For people living with dementia who are reported as missing to improve search effectiveness by better defining areas where police should search.\n\n###\n\n First published 22 November 2016  Updated 15 March 2023   Latest changes  Written by College of Policing  Missing persons  30 mins read   Implications for the UK leaving the European Union are currently under review – please see\xa0APP\xa0on international investigation\xa0for latest available detail on specific areas, for example: Schengen Information System Europol INTERPOL Joint Investigation Teams This section provides additional information to aid the investigation based on the vulnerability of the individual and the circumstances in which they are missing. Missing children Safeguarding young and vulnerable people is a responsibility of the police service and partner agencies (see\xa0Children Act 2004). When the police are notified that a child is missing, there is a clear responsibility on them to prevent the child from coming to harm. Where appropriate, a strategy meeting may be held. For further information see: Voice of the child\xa0 Voice of the child practice briefing\xa0 Section 11 of the Children Act 2004 Department for Education (2014) Statutory guidance on children who run away or go missing from home or care Children’s Views on being Reported Missing from Care Young people and risky behaviour Children and young people often do not have the same levels of awareness or ability to keep themselves safe as adults. Going missing may indicate that something is wrong in their lives. Many of the children and young people who repeatedly go missing are considered by some to be ‘streetwise’ and able to look after themselves. However, these children may not understand the risk they are exposing themselves to, and should not be treated as low/no apparent risk simply due to their apparent willingness/complicity. Children\xa0may put themselves in danger because they may have been abused, neglected or rejected by their families or others and,\xa0as a result, they may engage in further risky behaviours, such as: misusing substances committing crimes having risky sexual contacts living on the streets mixing with inappropriate adults Information relevant to the child When a missing person report relates to a looked-after child, it is important to\xa0work with all the agencies and carers\xa0that\xa0have been\xa0in regular contact with the child as they\xa0may have information about the child that might help to locate them. When a child is missing from care, close engagement with the carers is important.\n\n###\n\nThreshold for referral of missing persons: the individual is a ‘repeat missing person’, (reported as missing three times in a rolling 90 day period) the individual has experienced, or is likely to experience significant harm for children, the parent or carer appears unable or unwilling to work to support and meet the needs of a child that has gone missing. The\xa0Protection Procedures\xa0also recommend that there is very close working between children's social care and policing to ensure that all investigations are undertaken efficiently and without duplication of effort. Where appropriate, a multi-agency meeting is convened after a child has been missing from home or care for more than seven days, or has been missing on more than three occasions in a twelve month period. While standard thresholds for referral may be useful, senior officers will want to be sure that a process is in place to ensure cases are recognised which may require a greater safeguarding response before the threshold has been reached. For example, individuals reported missing for the first time where significant risks have been identified should be referred immediately for multi-agency support, without requiring further reports. For further information see\xa0HM Government (2018) Working together to safeguard children: A guide to inter-agency working to safeguard and promote the welfare of children. Prevention and intervention strategies Collecting and analysing data about cases will help the police and other agencies to understand whether there are any patterns related to missing persons incidents. The results of routine data analysis should be shared to inform the development and review of prevention and intervention strategies. Missing Persons Coordinators are vital for this information analysis and sharing. Interventions might include, for example, the use of\xa0Child Abduction Warning Notices, or referrals to support services. It is important that such interventions take place in appropriate circumstances and are not used as a single response when a person is at risk of harm, but form one strand of a more comprehensive approach. Regular liaison between neighbourhood policing teams and children’s and adults’ care providers may enable relationships to be developed between police officers, the staff of care establishments, and individuals who are looked after. These relationships may then support effective police intervention and the speedy resolution of cases. For lessons from other cases, see\xa0IOPC\xa0Learning the Lesson. Understanding the reasons for going missing Understanding the reasons why an individual went missing may help to prevent future harm to those individuals. The officers in charge of local areas should have clear plans on how they intend to reduce the number of people who go missing in their area.\n\n###\n\nIt’s also been raining heavily in the night and we have further calls about flooding in the road, so we ring Highways to inform them. I have a little smile to myself as I remember a call in the summer about cars stopping on the M11 because a mother duck and her ducklings were crossing the road. Lunchtime looms. I’m feeling hungry, but that disappears when I take a call from a 16-year-old male, who tells me that he can’t cope any more. He has cut himself with a knife but he doesn’t want to die. His sister has just had a baby. This goes on an emergency straight away, and officers are dispatched within three minutes. I have to talk to him about anything I can to distract him from his misery – luckily, I am good at small talk! Officers arrive and I feel relief as I can hang up the phone. COVID-19 has really affected Essex this year. People are low and weary. You can hear it in their voices. The number of mental health incidents has gone through the roof, and even the force control room team is quieter. Because of\xa0the onset of lockdown restrictions, more calls are coming in from the public reporting their neighbours for flouting the rules: 'We’re following the rules, why don’t they? What makes them think they are special?'\xa0 Gone are the past calls about drunken people leaving the pub. Instead, we have members of the public who are tired of being tied to the house and resentful of those who ignore the restrictions. After lunch, we receive a flurry of calls. There’s a domestic, involving a woman who tells me that ‘he didn’t mean to hit me, he loves me’. I spend time with this caller. There are three horses in the road. A driver has hit a dog and is upset, so I reassure him it wasn’t his fault. It gets busier. Essex is up and running but I am not. I feel tired but this is my job, so I make sure that nobody will hear it in my voice. Finally, it’s time to go home and hang up the headset for another day. I tend not to reflect on my day too much, so I can have some time to myself. There is no typical day in the control room.")

about_james = """ 
James is a 34 year old man, who was reported missing by his wife this evening as he has not returned home from work. It is now 2200, and she expected him home by 1900.
She says while he does go out for drinks after work sometimes, he has not been out this late before, and his phone is off.
James is in good health, there are no mental health concerns or other vulnerabilities. The weather is good, and his friend from work said he'd probably just gone out for drinks.
"""

james_answer = machine_risk_assessment(about_james, df, debug=True)
james_answer

Question:
 
James is a 34 year old man, who was reported missing by his wife this evening as he has not returned home from work. It is now 2200, and she expected him home by 1900.
She says while he does go out for drinks after work sometimes, he has not been out this late before, and his phone is off.
James is in good health, there are no mental health concerns or other vulnerabilities. The weather is good, and his friend from work said he'd probably just gone out for drinks.

Context:
 First published 22 November 2016  Updated 15 March 2023   Latest changes  Written by College of Policing  Missing persons  30 mins read   Implications for the UK leaving the European Union are currently under review – please see APP on international investigation for latest available detail on specific areas, for example: Schengen Information System Europol INTERPOL Joint Investigation Teams This section provides additional information to aid the investigation based on the vulnerability of the individual and the circumstances in which they are missing. Missing children Safeguarding young and vulnerable people is a responsibility of the police service and partner agencies (see Children Act 2004). When the police are notified that a child is missing, there is a clear responsibility on them to prevent the child from coming to harm. Where appropriate, a strategy meeting may be held. For further information see: Voice of the child  Voice of the child practice briefing  Section 11 of the Children Act 2004 Department for Education (2014) Statutory guidance on children who run away or go missing from home or care Children’s Views on being Reported Missing from Care Young people and risky behaviour Children and young people often do not have the same levels of awareness or ability to keep themselves safe as adults. Going missing may indicate that something is wrong in their lives. Many of the children and young people who repeatedly go missing are considered by some to be ‘streetwise’ and able to look after themselves. However, these children may not understand the risk they are exposing themselves to, and should not be treated as low/no apparent risk simply due to their apparent willingness/complicity. Children may put themselves in danger because they may have been abused, neglected or rejected by their families or others and, as a result, they may engage in further risky behaviours, such as: misusing substances committing crimes having risky sexual contacts living on the streets mixing with inappropriate adults Information relevant to the child When a missing person report relates to a looked-after child, it is important to work with all the agencies and carers that have been in regular contact with the child as they may have information about the child that might help to locate them. When a child is missing from care, close engagement with the carers is important.

###

 First published 22 November 2016  Updated 16 February 2023   Latest changes  Written by College of Policing  Missing persons  11 mins read   Introduction Going missing should be treated as an indicator that the individual may be at risk of harm. The safeguarding of vulnerable people is paramount and a missing person report should be recognised as an opportunity to identify and address risks. The reasons for a person deciding to go missing may be complex and linked to a variety of social or family issues. Three key factors should be considered in a missing person investigation: protecting those at risk of harm minimising distress and ensuring high quality of service to the families and carers of missing persons prosecuting those who perpetrate harm or pose a risk of harm when this is appropriate and supported by evidence Support for law enforcement agencies Police investigators can contact the following specialists for advice and assistance in missing and unidentified person investigations. UK Missing Persons Unit (UKMPU) on 0800 234 6034 NCA Major Crime Investigative Support (MCIS) on 0345 000 5463 Definition of ‘missing’ Anyone whose whereabouts cannot be established will be considered as missing until located, and their well-being or otherwise confirmed. All reports of missing people sit within a continuum of risk from ‘no apparent risk (absent)’ through to high-risk cases that require immediate, intensive action.  Risk assessment and response The risk assessment table The following table should be used as a guide to an appropriate level of police response based on initial and on-going risk assessment in each case. Risk assessment should be guided by the College of Policing Risk principles, the National Decision Model and Police Code of Ethics. No apparent risk (absent) There is no apparent risk of harm to either the subject or the public. Actions to locate the subject and/or gather further information should be agreed with the informant and a latest review time set to reassess the risk. Low risk The risk of harm to the subject or the public is assessed as possible but minimal. Proportionate enquiries should be carried out to ensure that the individual has not come to harm. Medium risk The risk of harm to the subject or the public is assessed as likely but not serious. This category requires an active and measured response by the police and other agencies in order to trace the missing person and support the person reporting.

###

High risk The risk of serious harm to the subject or the public is assessed as very likely. This category almost always requires the immediate deployment of police resources – action may be delayed in exceptional circumstances, such as searching water or forested areas during hours of darkness. A member of the senior management team must be involved in the examination of initial lines of enquiry and approval of appropriate staffing levels. Such cases should lead to the appointment of an investigating officer (IO) and possibly an SIO, and a police search adviser (PolSA).              There should be a press/media strategy and/or close contact with outside agencies. Family support should be put in place where appropriate. The UKMPU should be notified of the case without undue delay. Children’s services must also be notified immediately if the person is under 18. Risk of serious harm has been defined as (Home Office 2002 and OASys 2006): A risk which is life threatening and/or traumatic, and from which recovery, whether physical or psychological, can be expected to be difficult or impossible. Where the risk cannot be accurately assessed without active investigation, appropriate lines of enquiry should be set to gather the required information to inform the risk assessment. The missing persons process chart  Joint responsibility The police are entitled to expect parents and carers, including staff acting in a parenting role in care homes, to accept normal parenting responsibilities and undertake reasonable actions to try and establish the whereabouts of the individual. Children who are breaching parental discipline should not be dealt with by police unless there are other risks. For example, a child who is late home from a party should not be regarded as missing until the parent or carer has undertaken enquiries to locate the child. Once those enquiries have been completed, it may be appropriate to record the child as missing and take actions set out in this APP. Parents or carers may need police support if they are very distressed, incapacitated or otherwise unable to undertake enquiries. In such circumstances, it may be appropriate to make a referral to the local authority so that the standard of care for the missing person can be reviewed. Individuals whose whereabouts are known will not be considered as missing, but may require other police activity in order to ensure their welfare. Police should consult their local public protection procedures to ensure an appropriate safeguarding response is provided.

('Graded as Medium risk, because of the below risk factors:\n- James has not returned home at the expected time, and his phone is off\n- There is no indication of any mental health concerns or other vulnerabilities that could put him at risk',
 ' First published 22 November 2016  Updated 15 March 2023   Latest changes  Written by College of Policing  Missing persons  30 mins read   Implications for the UK leaving the European Union are currently under review – please see\xa0APP\xa0on international investigation\xa0for latest available detail on specific areas, for example: Schengen Information System Europol INTERPOL Joint Investigation Teams This section provides additional information to aid the investigation based on the vulnerability of the individual and the circumstances in which they are missing. Missing children Safeguarding young and vulnerable people is a responsibility of the police service and partner agencies (see\xa0Children Act 2004). When the police are notified that a child is missing, there is a clear responsibility on them to prevent the child from coming to harm. Where appropriate, a strategy meeting may be held. For further information see: Voice of the child\xa0 Voice of the child practice briefing\xa0 Section 11 of the Children Act 2004 Department for Education (2014) Statutory guidance on children who run away or go missing from home or care Children’s Views on being Reported Missing from Care Young people and risky behaviour Children and young people often do not have the same levels of awareness or ability to keep themselves safe as adults. Going missing may indicate that something is wrong in their lives. Many of the children and young people who repeatedly go missing are considered by some to be ‘streetwise’ and able to look after themselves. However, these children may not understand the risk they are exposing themselves to, and should not be treated as low/no apparent risk simply due to their apparent willingness/complicity. Children\xa0may put themselves in danger because they may have been abused, neglected or rejected by their families or others and,\xa0as a result, they may engage in further risky behaviours, such as: misusing substances committing crimes having risky sexual contacts living on the streets mixing with inappropriate adults Information relevant to the child When a missing person report relates to a looked-after child, it is important to\xa0work with all the agencies and carers\xa0that\xa0have been\xa0in regular contact with the child as they\xa0may have information about the child that might help to locate them. When a child is missing from care, close engagement with the carers is important.\n\n###\n\n First published 22 November 2016  Updated 16 February 2023   Latest changes  Written by College of Policing  Missing persons  11 mins read   Introduction Going missing should be treated as an indicator that the individual may be at risk of harm. The safeguarding of vulnerable people is paramount and a missing person report should be recognised as an opportunity to identify and address risks. The reasons for a person deciding to go missing may be complex and linked to a variety of social or family issues. Three key factors should be considered in a missing person investigation: protecting those at risk of harm minimising distress and ensuring high quality of service to the families and carers of missing persons prosecuting those who perpetrate harm or pose a risk of harm when this is appropriate and supported by evidence Support for law enforcement agencies Police investigators can contact the following specialists for advice and assistance in missing and unidentified person investigations. UK Missing Persons Unit (UKMPU) on 0800\xa0234 6034 NCA\xa0Major Crime Investigative Support (MCIS) on 0345 000 5463 Definition of ‘missing’ Anyone whose whereabouts cannot be established will be considered as missing until located, and their well-being or otherwise confirmed. All reports of missing people sit within a continuum of risk from ‘no apparent risk (absent)’ through to high-risk cases\xa0that require immediate, intensive action.  Risk assessment and response The risk assessment table The following table should be used as a guide to\xa0an appropriate level of police response based on initial and on-going risk assessment in each case. Risk assessment should be guided by the College of Policing\xa0Risk principles,\xa0the\xa0National Decision Model\xa0and Police\xa0Code of Ethics. No apparent risk (absent) There is no apparent risk of harm to either the subject or the public. Actions to locate the subject and/or gather further information should be agreed with the informant and a latest review time set to reassess the risk. Low risk The risk of harm to the subject or the public is assessed as possible but minimal. Proportionate enquiries should be carried out to ensure that the individual has not come to harm. Medium risk The risk of harm to the subject or the public is assessed as likely but not serious. This category requires an active and measured response by the police and other agencies in order to trace the missing person and support the person reporting.\n\n###\n\nHigh risk The risk of serious harm to the subject or the public is assessed as very likely. This category almost always requires the immediate deployment of police resources – action may be delayed in exceptional circumstances, such as searching water or forested areas during hours of darkness. A member of the senior management team must be involved in the examination of initial lines of enquiry and approval of appropriate staffing levels. Such cases should lead to the appointment of an investigating officer (IO) and possibly an\xa0SIO, and a police search adviser (PolSA). \t\t\t\xa0 There should be a press/media strategy and/or close contact with outside agencies. Family support should be put in place where appropriate. The UKMPU\xa0should be notified of the case without undue delay. Children’s services must also be notified immediately if the person is under 18. Risk of serious harm has been defined as (Home Office 2002 and OASys 2006): A risk which is life threatening and/or traumatic, and from which recovery, whether physical or psychological, can be expected to be difficult or impossible. Where the risk cannot be accurately assessed without active investigation, appropriate lines of enquiry should be set to gather the required information to inform the risk assessment. The missing persons process chart  Joint responsibility The police are entitled to expect parents and carers, including staff acting in a parenting role in care homes, to accept normal parenting responsibilities and undertake reasonable actions to try and establish the whereabouts of the individual. Children who are breaching parental discipline should not be dealt with by police unless there are other risks. For example, a child who is late home from a party should not be regarded as missing until the parent or carer has undertaken enquiries to locate the child. Once those enquiries have been completed, it may be appropriate to record the child as missing and take actions set out in this\xa0APP. Parents or carers may need police support if they are very distressed, incapacitated or otherwise unable to undertake enquiries. In such circumstances, it may be appropriate to make a referral to the local authority so that the standard of care for the missing person can be reviewed. Individuals whose whereabouts are known will not be considered as missing, but may require other police activity in order to ensure their welfare. Police should consult their local public protection procedures to ensure an appropriate safeguarding response is provided.')

about_yannik = """ Yannik is a 15 year old boy. He has recently been down, and was reported missing by his parents as he did not return home from school today.

His friends are worried he may be depressed, and when he apparently told one a few days ago 'if it doesn't get any better, I'm going to end it soon'
"""

yannik_answer = machine_risk_assessment(about_yannik, df, debug=True)
yannik_answer

Question:
 Yannik is a 15 year old boy. He has recently been down, and was reported missing by his parents as he did not return home from school today.

His friends are worried he may be depressed, and when he apparently told one a few days ago 'if it doesn't get any better, I'm going to end it soon'

Context:
 First published 22 November 2016  Updated 15 March 2023   Latest changes  Written by College of Policing  Missing persons  30 mins read   Implications for the UK leaving the European Union are currently under review – please see APP on international investigation for latest available detail on specific areas, for example: Schengen Information System Europol INTERPOL Joint Investigation Teams This section provides additional information to aid the investigation based on the vulnerability of the individual and the circumstances in which they are missing. Missing children Safeguarding young and vulnerable people is a responsibility of the police service and partner agencies (see Children Act 2004). When the police are notified that a child is missing, there is a clear responsibility on them to prevent the child from coming to harm. Where appropriate, a strategy meeting may be held. For further information see: Voice of the child  Voice of the child practice briefing  Section 11 of the Children Act 2004 Department for Education (2014) Statutory guidance on children who run away or go missing from home or care Children’s Views on being Reported Missing from Care Young people and risky behaviour Children and young people often do not have the same levels of awareness or ability to keep themselves safe as adults. Going missing may indicate that something is wrong in their lives. Many of the children and young people who repeatedly go missing are considered by some to be ‘streetwise’ and able to look after themselves. However, these children may not understand the risk they are exposing themselves to, and should not be treated as low/no apparent risk simply due to their apparent willingness/complicity. Children may put themselves in danger because they may have been abused, neglected or rejected by their families or others and, as a result, they may engage in further risky behaviours, such as: misusing substances committing crimes having risky sexual contacts living on the streets mixing with inappropriate adults Information relevant to the child When a missing person report relates to a looked-after child, it is important to work with all the agencies and carers that have been in regular contact with the child as they may have information about the child that might help to locate them. When a child is missing from care, close engagement with the carers is important.

###

It’s also been raining heavily in the night and we have further calls about flooding in the road, so we ring Highways to inform them. I have a little smile to myself as I remember a call in the summer about cars stopping on the M11 because a mother duck and her ducklings were crossing the road. Lunchtime looms. I’m feeling hungry, but that disappears when I take a call from a 16-year-old male, who tells me that he can’t cope any more. He has cut himself with a knife but he doesn’t want to die. His sister has just had a baby. This goes on an emergency straight away, and officers are dispatched within three minutes. I have to talk to him about anything I can to distract him from his misery – luckily, I am good at small talk! Officers arrive and I feel relief as I can hang up the phone. COVID-19 has really affected Essex this year. People are low and weary. You can hear it in their voices. The number of mental health incidents has gone through the roof, and even the force control room team is quieter. Because of the onset of lockdown restrictions, more calls are coming in from the public reporting their neighbours for flouting the rules: 'We’re following the rules, why don’t they? What makes them think they are special?'  Gone are the past calls about drunken people leaving the pub. Instead, we have members of the public who are tired of being tied to the house and resentful of those who ignore the restrictions. After lunch, we receive a flurry of calls. There’s a domestic, involving a woman who tells me that ‘he didn’t mean to hit me, he loves me’. I spend time with this caller. There are three horses in the road. A driver has hit a dog and is upset, so I reassure him it wasn’t his fault. It gets busier. Essex is up and running but I am not. I feel tired but this is my job, so I make sure that nobody will hear it in my voice. Finally, it’s time to go home and hang up the headset for another day. I tend not to reflect on my day too much, so I can have some time to myself. There is no typical day in the control room.

###

Police officers should ask the care home or local authority for details of the child’s risk assessment so that it can be taken into account during the investigation. Many forces are using the Philomena Protocol to guide their actions in relation to relevant cases involving children. A child, especially a teenager, is unlikely to share all information about their life with their parents or carers. Investigators should not overlook information from siblings, friends, associates, school teachers and others. The online activity of the child may also provide valuable additional information which parents and carers may not be aware of. For further information see: Children's views on being reported missing from care  No Place at Home - Risks facing children and young people who go missing from out of area placements Child Rescue Alert Child Rescue Alert (CRA) – (available to authorised users logged on to the restricted online College Learn) is a partnership between the police and the media which seeks public assistance when it is feared that a child may be at risk of serious harm. Assistance is sought via TV, radio, text messaging, social and digital media (including the internet) so that relevant information relating to the child, offender or any specified vehicle is passed on to the police. CRA focuses on the risk to the child, rather than whether or not an offence has taken place. The criteria for launching an alert is: The child is apparently under 18 years old There is a perception that the child is in imminent danger of serious harm or death There is sufficient information available to enable the public to assist police in locating the child Child Rescue Alert Activation Protocol (available to authorised users logged on to the restricted online College Learn) The CRA has been expanded to enable alerts to be disseminated by the charity Missing People. The system is managed by the National Crime Agency (NCA) and specialist advice is available 24/7 by contacting 0800 234 6034.

###

This is particularly true to the Greater Manchester area. As a result, this research seeks to utilise this appeal in providing a platform for young people who have received a Threats to Life Notice and children to parents who have received a Threat to Life Notice, in order to steer them away from potential criminal activity and receive mentoring through football coaching, education and employability skills. Operationally, this would seek to provide tactical options in the short and long-term in instances where a young person (under the age of 18) is particularly vulnerable as either a victim or perpetrator of criminal behaviour, and potentially reduce the threat, risk and harm pertaining to such individuals. The key aims are: to develop the individuals in the cohort with regards to qualifications, personal wellbeing, skill development, employability and social action (part of their local community and their perception of their community) – this in turn will potentially reduce the impact of trauma and increase confidence and self-esteem of the cohort to reduce the effects of adverse childhood experiences through re-sensitisation and raising awareness that adverse childhood experiences should not be considered the norm to understand and improve the delivery and impact of Threats to Life on young people to inform police forces on the best possible process and follow-on actions needed when delivering Threats to Life Notices  Research methodology A cohort of 16 young people between the ages of 13 to 18 will be identified through GMP’s systems who have either directly received a Threat to Life Notice in the past 12 months, or are children to a parent(s) who have received a Threat to Life Notice in the past 12 months. Their details will then be given to the mentors at the Manchester United Youth Foundation who then make contact through post in the first instance to invite the young person into the scheme. More individuals will be invited into the scheme as it progresses to ensure a minimum of 16 participants at any given stage (catering for any refusals to be involved or disengagement).

('Graded as High risk, because of the below risk factors: \n- Yannik is a 15 year old boy who has been reported missing by his parents\n- His friends are worried he may be depressed\n- He has expressed suicidal ideation to one of his friends',
 " First published 22 November 2016  Updated 15 March 2023   Latest changes  Written by College of Policing  Missing persons  30 mins read   Implications for the UK leaving the European Union are currently under review – please see\xa0APP\xa0on international investigation\xa0for latest available detail on specific areas, for example: Schengen Information System Europol INTERPOL Joint Investigation Teams This section provides additional information to aid the investigation based on the vulnerability of the individual and the circumstances in which they are missing. Missing children Safeguarding young and vulnerable people is a responsibility of the police service and partner agencies (see\xa0Children Act 2004). When the police are notified that a child is missing, there is a clear responsibility on them to prevent the child from coming to harm. Where appropriate, a strategy meeting may be held. For further information see: Voice of the child\xa0 Voice of the child practice briefing\xa0 Section 11 of the Children Act 2004 Department for Education (2014) Statutory guidance on children who run away or go missing from home or care Children’s Views on being Reported Missing from Care Young people and risky behaviour Children and young people often do not have the same levels of awareness or ability to keep themselves safe as adults. Going missing may indicate that something is wrong in their lives. Many of the children and young people who repeatedly go missing are considered by some to be ‘streetwise’ and able to look after themselves. However, these children may not understand the risk they are exposing themselves to, and should not be treated as low/no apparent risk simply due to their apparent willingness/complicity. Children\xa0may put themselves in danger because they may have been abused, neglected or rejected by their families or others and,\xa0as a result, they may engage in further risky behaviours, such as: misusing substances committing crimes having risky sexual contacts living on the streets mixing with inappropriate adults Information relevant to the child When a missing person report relates to a looked-after child, it is important to\xa0work with all the agencies and carers\xa0that\xa0have been\xa0in regular contact with the child as they\xa0may have information about the child that might help to locate them. When a child is missing from care, close engagement with the carers is important.\n\n###\n\nIt’s also been raining heavily in the night and we have further calls about flooding in the road, so we ring Highways to inform them. I have a little smile to myself as I remember a call in the summer about cars stopping on the M11 because a mother duck and her ducklings were crossing the road. Lunchtime looms. I’m feeling hungry, but that disappears when I take a call from a 16-year-old male, who tells me that he can’t cope any more. He has cut himself with a knife but he doesn’t want to die. His sister has just had a baby. This goes on an emergency straight away, and officers are dispatched within three minutes. I have to talk to him about anything I can to distract him from his misery – luckily, I am good at small talk! Officers arrive and I feel relief as I can hang up the phone. COVID-19 has really affected Essex this year. People are low and weary. You can hear it in their voices. The number of mental health incidents has gone through the roof, and even the force control room team is quieter. Because of\xa0the onset of lockdown restrictions, more calls are coming in from the public reporting their neighbours for flouting the rules: 'We’re following the rules, why don’t they? What makes them think they are special?'\xa0 Gone are the past calls about drunken people leaving the pub. Instead, we have members of the public who are tired of being tied to the house and resentful of those who ignore the restrictions. After lunch, we receive a flurry of calls. There’s a domestic, involving a woman who tells me that ‘he didn’t mean to hit me, he loves me’. I spend time with this caller. There are three horses in the road. A driver has hit a dog and is upset, so I reassure him it wasn’t his fault. It gets busier. Essex is up and running but I am not. I feel tired but this is my job, so I make sure that nobody will hear it in my voice. Finally, it’s time to go home and hang up the headset for another day. I tend not to reflect on my day too much, so I can have some time to myself. There is no typical day in the control room.\n\n###\n\nPolice officers should ask the care home or local authority for details of the child’s risk assessment so that it can be taken into account during the investigation.\xa0Many forces are using the Philomena Protocol to guide their actions in relation to relevant cases involving children. A child, especially a teenager, is unlikely to share all information about their life with their parents or carers. Investigators should not overlook information from siblings, friends, associates, school teachers and others. The online activity of the child may also provide valuable additional information which parents and carers may not be aware of. For further information see: Children's views on being reported missing from care\xa0 No Place at Home - Risks facing children and young people who go missing from out of area placements Child Rescue Alert Child Rescue Alert\xa0(CRA) – (available to authorised users logged on to the restricted online\xa0College Learn) is a partnership between the police and the media which seeks public assistance when it is feared that a child may be at risk of serious harm. Assistance is sought via TV, radio, text messaging, social and digital media (including the internet) so that relevant information relating to the child, offender or any specified vehicle is passed on to the police. CRA\xa0focuses on the risk to the child, rather than whether or not an offence has taken place. The criteria for launching an alert is: The child is apparently under 18 years old There is a perception that the child is in imminent danger of serious harm or death There is sufficient information available to enable the public to assist police in locating the child Child Rescue Alert Activation Protocol\xa0(available to authorised users logged on to the restricted online\xa0College Learn) The\xa0CRA\xa0has been expanded to enable alerts to be disseminated by the charity Missing People. The system is managed by the National Crime\xa0Agency (NCA) and specialist advice is available 24/7 by contacting 0800\xa0234 6034.\n\n###\n\nThis is particularly true to the Greater Manchester area. As a result, this research seeks to utilise this appeal in providing a platform for young people who have received a Threats to Life Notice and children to parents who have received a Threat to Life Notice, in order to steer them away from potential criminal activity and receive mentoring through football coaching, education and employability skills. Operationally, this would seek to provide tactical options in the short and long-term in instances where a young person (under the age of 18) is particularly vulnerable as either a victim or perpetrator of criminal behaviour, and potentially reduce the threat, risk and harm pertaining to such individuals. The key aims are: to develop the individuals in the cohort with regards to qualifications, personal wellbeing, skill development, employability and social action (part of their local community and their perception of their community) – this in turn will potentially reduce the impact of trauma and increase confidence and self-esteem of the cohort to reduce the effects of adverse childhood experiences through re-sensitisation and raising awareness that adverse childhood experiences should not be considered the norm to understand and improve the delivery and impact of Threats to Life on young people to inform police forces on the best possible process and follow-on actions needed when delivering Threats to Life Notices  Research methodology A cohort of 16 young people between the ages of 13 to 18 will be identified through GMP’s systems who have either directly received a Threat to Life Notice in the past 12 months, or are children to a parent(s) who have received a Threat to Life Notice in the past 12 months. Their details will then be given to the mentors at the Manchester United Youth Foundation who then make contact through post in the first instance to invite the young person into the scheme. More individuals will be invited into the scheme as it progresses to ensure a minimum of 16 participants at any given stage (catering for any refusals to be involved or disengagement).")

jason_risk_profile = """ Jason is a 15 year old adult male, who has gone missing from his care home in Southwark. His carer has contacted the school, which has said he was not in today.
They that this is not the first time, and that Jason has been seen hanging out with older boys, who may be involved in crime and drugs."""


jason_answer = machine_risk_assessment(jason_risk_profile, df, debug=True)
jason_answer

Question:
 Jason is a 15 year old adult male, who has gone missing from his care home in Southwark. His carer has contacted the school, which has said he was not in today.
They that this is not the first time, and that Jason has been seen hanging out with older boys, who may be involved in crime and drugs.
Context:
 First published 22 November 2016  Updated 15 March 2023   Latest changes  Written by College of Policing  Missing persons  30 mins read   Implications for the UK leaving the European Union are currently under review – please see APP on international investigation for latest available detail on specific areas, for example: Schengen Information System Europol INTERPOL Joint Investigation Teams This section provides additional information to aid the investigation based on the vulnerability of the individual and the circumstances in which they are missing. Missing children Safeguarding young and vulnerable people is a responsibility of the police service and partner agencies (see Children Act 2004). When the police are notified that a child is missing, there is a clear responsibility on them to prevent the child from coming to harm. Where appropriate, a strategy meeting may be held. For further information see: Voice of the child  Voice of the child practice briefing  Section 11 of the Children Act 2004 Department for Education (2014) Statutory guidance on children who run away or go missing from home or care Children’s Views on being Reported Missing from Care Young people and risky behaviour Children and young people often do not have the same levels of awareness or ability to keep themselves safe as adults. Going missing may indicate that something is wrong in their lives. Many of the children and young people who repeatedly go missing are considered by some to be ‘streetwise’ and able to look after themselves. However, these children may not understand the risk they are exposing themselves to, and should not be treated as low/no apparent risk simply due to their apparent willingness/complicity. Children may put themselves in danger because they may have been abused, neglected or rejected by their families or others and, as a result, they may engage in further risky behaviours, such as: misusing substances committing crimes having risky sexual contacts living on the streets mixing with inappropriate adults Information relevant to the child When a missing person report relates to a looked-after child, it is important to work with all the agencies and carers that have been in regular contact with the child as they may have information about the child that might help to locate them. When a child is missing from care, close engagement with the carers is important.

###

These include children or young people who: go missing – especially on regular occasions from home or care live in a chaotic or dysfunctional family have a history of domestic abuse within the family environment have a history of abuse (including familial child sexual abuse, risk of forced marriage, risk of honour-based violence, physical and emotional abuse and neglect) have experienced or are experiencing problematic parenting have parents who misuse drugs or alcohol have parents with health problems are young carers within the family unit experience social exclusion as a result of poverty have experienced recent bereavement or loss have unsupervised use of social networking chat rooms/sites have mental ill health have social or learning difficulties have low self-esteem or self-confidence are unsure about their sexual orientation or are unable to confide in their family about their sexual orientation misuse alcohol and/or drugs have been or are excluded from mainstream education are involved in gang activity attend school with other young people who are sexually exploited are friends with individuals who are sexually exploited do not have friends in the same age group are being bullied live in care, foster care, hostels and/or bed and breakfast accommodation – particularly when living out of their home area are homeless have associations with gangs through relatives, peers or intimate relationships live in a gang neighbourhood This is not an exhaustive list, nor have the vulnerabilities been listed in order of importance. Children from loving and secure homes can also be victims of sexual exploitation. The characteristics common to all victims are not always their age, ethnicity, disability or sexual orientation, but their powerlessness and vulnerability. Warning signs Despite the increased profile of CSE and improvements in how the police work with partner agencies, CSE cases are still under-reported. The Office of the Children’s Commissioner conducted a two-year inquiry into child sexual exploitation in gangs and groups. Their 2013 report, If only someone had listened, highlights that sexually exploited children are not always identified even when they show signs of being victims. Numerous warning signs were identified in the Office of the Children’s Commissioner 2012 interim report, I thought I was the only one, the only one in the world, which can indicate that a young person is being forced or manipulated into sexual activity and is a victim of sexual exploitation. Practitioners need to be aware of these warning signs and recognise that a victim does not have to exhibit all of the warning signs to be a victim of sexual exploitation.

###

Staff should tell the detainee that she can ask to see the carer at any time. Forces must implement policies and procedures to ensure that all girls who are detained and in custody are under the care of a woman. For further information, see PACE Code C, section 3.20A. Transporting children and young people Children and young persons under the age of 18 are not allowed to associate with adult detainees while being detained, conveyed to and from court or waiting to be so conveyed. An exception to this is permitted in accordance with the Children and Young Persons Act 1933 section 31, where the young person is jointly charged with an adult or relative. Officers should make arrangements to prevent association when the child or young person is: detained in a police station being conveyed to or from any criminal court attending court Children and young people should not be carried in a vehicle with adult detainees unless the vehicle has been designed and built to carry them separately and simultaneously. Vehicles that are available for this specific purpose are authorised under the 2011 Prisoner Escort and Custody Services’ contract arrangements. Appropriate adults Forces should establish policies and protocols to provide access to appropriate adults for young persons in police custody. Local YOTs have a statutory responsibility to ensure that an appropriate adult service is provided for children and young people, whether they provide the service themselves or contract a voluntary or private sector agency to do so on their behalf. It is the responsibility of the appropriate adult provider to work with the local force to develop policies and protocols to ensure there is effective provision of appropriate adult services in line with Youth Justice Board (2014) case management guidance. This guidance makes it clear that the appropriate adult service should operate out of hours as well as during standard working hours. All appropriate adults, custody managers, custody officers and staff must be aware of their role as defined by PACE, and also of any agreed local policies, protocols or service level agreements for providing appropriate adults. For further information, see PACE and section 38 of the Crime and Disorder Act 1998. See also the Youth Justice Board’s (YJB) National Standards for Youth Justice Services. The YJB and National Appropriate Adult Network have also published joint guidance and advice on appropriate adult services. When should an appropriate adult be contacted? Detention can be very stressful so it is important that an appropriate adult attends as soon as is practicable to minimise the amount of time the child or young person spends in detention.

('Graded as Medium risk, because of the below risk factors: \n- Jason is a 15 year old adult male, who has gone missing from his care home in Southwark\n- This is not the first time he has gone missing\n- He has been seen hanging out with older boys, who may be involved in crime and drugs',
 ' First published 22 November 2016  Updated 15 March 2023   Latest changes  Written by College of Policing  Missing persons  30 mins read   Implications for the UK leaving the European Union are currently under review – please see\xa0APP\xa0on international investigation\xa0for latest available detail on specific areas, for example: Schengen Information System Europol INTERPOL Joint Investigation Teams This section provides additional information to aid the investigation based on the vulnerability of the individual and the circumstances in which they are missing. Missing children Safeguarding young and vulnerable people is a responsibility of the police service and partner agencies (see\xa0Children Act 2004). When the police are notified that a child is missing, there is a clear responsibility on them to prevent the child from coming to harm. Where appropriate, a strategy meeting may be held. For further information see: Voice of the child\xa0 Voice of the child practice briefing\xa0 Section 11 of the Children Act 2004 Department for Education (2014) Statutory guidance on children who run away or go missing from home or care Children’s Views on being Reported Missing from Care Young people and risky behaviour Children and young people often do not have the same levels of awareness or ability to keep themselves safe as adults. Going missing may indicate that something is wrong in their lives. Many of the children and young people who repeatedly go missing are considered by some to be ‘streetwise’ and able to look after themselves. However, these children may not understand the risk they are exposing themselves to, and should not be treated as low/no apparent risk simply due to their apparent willingness/complicity. Children\xa0may put themselves in danger because they may have been abused, neglected or rejected by their families or others and,\xa0as a result, they may engage in further risky behaviours, such as: misusing substances committing crimes having risky sexual contacts living on the streets mixing with inappropriate adults Information relevant to the child When a missing person report relates to a looked-after child, it is important to\xa0work with all the agencies and carers\xa0that\xa0have been\xa0in regular contact with the child as they\xa0may have information about the child that might help to locate them. When a child is missing from care, close engagement with the carers is important.\n\n###\n\nThese include children or young people who: go missing\xa0– especially on regular occasions from home or care live in a chaotic or dysfunctional family have a history of domestic abuse within the family environment have a history of abuse (including familial\xa0child sexual abuse, risk of\xa0forced marriage, risk of\xa0honour-based violence,\xa0physical\xa0and\xa0emotional\xa0abuse and\xa0neglect) have experienced or are experiencing problematic parenting have parents who misuse drugs or alcohol have parents with health problems are young carers within the family unit experience social exclusion as a result of poverty have experienced recent bereavement or loss have unsupervised use of social networking chat rooms/sites have mental ill health have social or learning difficulties have low self-esteem or self-confidence are unsure about their sexual orientation or are unable to confide in their family about their sexual orientation misuse alcohol and/or drugs have been or are excluded from mainstream education are involved in gang activity attend school with other young people who are sexually exploited are friends with individuals who are sexually exploited do not have friends in the same age group are being bullied live in care, foster care, hostels and/or bed and breakfast accommodation – particularly when living out of their home area are homeless have associations with gangs through relatives, peers or intimate relationships live in a gang neighbourhood This is not an exhaustive list, nor have the vulnerabilities been listed in order of importance. Children from loving and secure homes can also be victims of sexual exploitation. The\xa0characteristics common to all victims are not always their age, ethnicity, disability or sexual orientation, but their powerlessness and vulnerability. Warning signs Despite the increased profile of\xa0CSE\xa0and improvements in how the police work with partner agencies,\xa0CSE\xa0cases are still under-reported. The Office of the Children’s Commissioner conducted a two-year inquiry into child sexual exploitation in gangs and groups. Their 2013\xa0report,\xa0If only someone had listened, highlights that sexually exploited children are not always identified even when they show signs of being victims. Numerous warning signs were identified in the Office of the Children’s Commissioner 2012 interim report,\xa0I thought I was the only one, the only one in the world, which can indicate that\xa0a young person is being forced or manipulated into sexual activity and is a victim of sexual exploitation. Practitioners need to be aware of these warning signs and recognise that a victim does not\xa0have to exhibit all of the warning signs to be a victim of sexual exploitation.\n\n###\n\nStaff should tell the detainee that she can ask to see the carer at any time. Forces must implement policies and procedures to ensure that all girls who are detained and in custody are under the care of a woman. For further information, see\xa0PACE Code C, section 3.20A. Transporting children and young people Children and young persons under the age of 18 are not allowed to associate with adult detainees while being detained, conveyed to and from court or waiting to be so conveyed. An exception to this is permitted in accordance with the Children and Young Persons Act 1933 section 31, where the young person is jointly charged with an adult or relative. Officers should make arrangements to prevent association when the child or young person is: detained in a police station being conveyed to or from any criminal court attending court Children and young people should not be carried in a vehicle with adult detainees unless the vehicle has been designed and built to carry them separately and simultaneously. Vehicles that are available for this specific purpose are authorised under the 2011 Prisoner Escort and Custody Services’ contract arrangements. Appropriate adults Forces should establish policies and protocols to provide access to appropriate adults for young persons in police custody. Local YOTs have a statutory responsibility to ensure that an appropriate adult service is provided for children and young people, whether they provide the service themselves or contract a voluntary or private sector agency to do so on their behalf. It is the responsibility of the appropriate adult provider to work with the local force to develop policies and protocols to ensure there is effective provision of appropriate adult services in line with Youth Justice Board (2014) case management guidance. This guidance makes it clear that the appropriate adult service should operate out of hours as well as during standard working hours. All appropriate adults, custody managers, custody officers and staff must be aware of their role as defined by PACE, and also of any agreed local policies, protocols or service level agreements for providing appropriate adults. For further information, see PACE and section 38 of the Crime and Disorder Act 1998. See also the Youth Justice Board’s (YJB) National Standards for Youth Justice Services. The YJB and National Appropriate Adult Network have also published joint guidance and advice on appropriate adult services. When should an appropriate adult be contacted? Detention can be very stressful so it is important that an appropriate adult attends as soon as is practicable to minimise the amount of time the child or young person spends in detention.')

It’s honestly not too bad! while this certainly wouldn’t replace human decision making, it could certainly act as a safeguard against missing a key piece of information at three in the morning when you have a queue of 8 missing people to evaluate, each pages worth of history.

There are certainly some pretty major niggles you’d need to fix: for one I’m not sure how the model will perform given actual legislation, rather than “plain English” guidance, nor do I imagine it will cope particularly well with policing specific terminology. The risks it’s identified so far are all mostly obvious, rather than identifying one needle in a giant haystack of intelligence reports. It’s also all going through the OpenAI black-box servers, though I imagine that could be replaced with something open-ish like Llama without too much effort. But when I consider where NLP was even 18 months ago, and just how much computing cognitive power we’ve been able to deploy in only a few hours…who knows where we’ll be next year?

The hackers aren’t excited by AI…

Now, this is far from universal given CCC are a quite distinct, very German subset of the hacker community (and DEFCON last week definitely had some LLM excitement), but the lack of LLM talks in the schedule was stark… and and when they were mentioned, it was mostly critical. In hindsight, this isn’t so surprising given actual open-source LLMs essentially don’t exist and are dominated by US behemoths.

… but they’re still excited by SPACE

Honestly, I was not ready for the amount of excitement about space. I’d mostly forgotten Starlink was a thing, but there was real enthusiasm for the impact of reduced cheap satelite launches. Given I know bugger all about space launches, microsats and so on, it was striking to meet more people who wanted to talk about Planet, OneWeb and Relativity Space than OpenAI.

Twitter is dead, long live the Fediverse

I moved over to Mastodon earlier this year, but it’s at times been tough to use it as my primary social media given how few of my colleagues have followed…annoyingly, Twitter still served up more relevant content, more frequently. Not so at camp: #CCCamp23 was buzzing during the entire event. Again, I’m not sure how much of this comes down to Europe versus the US (with the German government hosting it’s own server) but especially given just how much the AI and machine learning communities remained glue to the birdsite, this was striking.

People are angry about digital identity

Now, yes, Germany is especially privacy focused (and I still think paying for everything with cash is mad), but European efforts to introduce digital identity got a fair few mentions, and never in a good way. CCCamp is a broad church, with villages of techno-communists a few minutes walk from crypto-libertarians, but all of them were concerned by formalised state intrusion into the digital space.

Closed systems are still insecure

Okay, this one is a little niche (I needed a list of 5, sue me) but one of my highlight talks was All Cops Are Broadcasting, discussing how security researchers had compromised the TETRA (Airwave in the UK) communications system used by most police forces. I’ve never really thought much about the architecture behind Airwave, but turns out it relies on a cryptographic “secret sauce”. This is never a good way of designing something secure: if your software/algorithm/cryptography relies on obscurity to stay safe, it will eventually become unsafe. So I guess it’s a good thing that the Airwave replacement is right around the corner.

Spatial Data

We begin by importing our spatial border data. Spatial coordinates are just coordinates, so without understanding what those coordinates mean (for instance, where you are in a city, or in the world, at what altitude, etc), they’re essentially points on a chart.

For us, this is provided by the Mayor’s Office for Policing and Crime, and also conveniently contains some area characteristics. We use Geopandas to read the file.

Geospatial modelling relies on assigning events to a unit of space. You could theoretically make this as detailed as possible - for instance, meter squares - but given we’re going to analyse how our units are interconnected, that’s probably not computationally feasible (if everything is connected to everything, you’re going to need a really big computer). You’ll need to reach a suitable compromise. Helpfully, the UK government provides various statistical units, including border coordinates, for download. Lower Super Output Areas (LSOAs) and Middle Super Output Areas (MSOAs) contain populations of between 5,000 and 7,200, and as such should be (partly) comparable.

Geospatial data will use a specific Coordinate Reference System, or CRS which will affect how your data is processed. Make sure you’re using the right one - worldwide data that assumes it is on a sphere will behave very differently to data from a specific country on a flat plane.

UK policing data often uses National Grid Easting/Northing coordinates, rather than the more common Lat/Longs. Daata.Police.uk comes pre-processed into lat/long coordinates. Thankfully, whichever you have, geopandas can easily convert (or “re-project”) to another system.

msoas = geopandas.read_file("statistical-gis-boundaries-london/MapInfo/MSOA_2011_London_gen_MHW.tab")
msoas = msoas.to_crs(epsg=4326)

C:\Users\Admin\Anaconda3\envs\geospatial\lib\site-packages\geopandas\geodataframe.py:422: RuntimeWarning: Sequential read of iterator was interrupted. Resetting iterator. This can negatively impact the performance.
  for feature in features_lst:

Our data on both stop and search and robberies was manually downloaded from data.police.uk, and extracted into our working folder. Given this is then returned into a folder per month, I have written a series of helper functions to read the files, assign them to an MSOA, and combine them into a total per MSOA.

To assign to an MSOA, we use a “spatial join”: like a normal table join (think vlookup in Excel), this connects elements from one table, to elements from another, via a common column. In our case, the common column is the geographic location: which MSOA is our search/crime located in.

Crime data from data.police.uk is separated into “major” crime types, which is very helpful for anonymisation (we can’t figure out who the victim was if we don’t know specific crime types), but does mean that all violent and sexual crime is agglomerated into one - given I think it’s unlikely searches deter sexual offences, that’s unhelpful. As such, we’ll focus on robbery, which is disaggregated. This isn’t ideal, but robbery remains a serious, violent, high priority crime, and as such you’d hope they are connected, and robbery might in fact be a rough proxy for overall violence.

def read_and_add(file_path):
    df = pd.read_csv(file_path)
    robberies = df[df["Crime type"] == 'Robbery']
    
    #we create a geopandas object that uses the lat/long coordinates, using the appropriate CRS code.
    robberies = geopandas.GeoDataFrame(robberies, geometry=geopandas.points_from_xy(robberies.Longitude, robberies.Latitude), crs="epsg:4326")
    detected_mask = robberies["Last outcome category"].isin(positive_outcomes)
    
    # identify which crimes resulted in a detected outcome 
    robberies.loc[detected_mask, "Detected"] = 1
    detected_mask = robberies["Last outcome category"].isin(positive_outcomes)

    detected_df = robberies[detected_mask]
    robberies.loc[detected_mask, "Detected"] = 1

    dfsjoin_r = geopandas.sjoin(msoas, robberies) #Spatial join Points to polygons
    dfsjoin_d = geopandas.sjoin(msoas, detected_df) #Spatial join Points to polygons

    #aggregate to a pivot table that will count our outputs.
    pivot_r = pd.pivot_table(dfsjoin_r,index='MSOA11CD',fill_value=0,columns='Crime type',aggfunc={'Crime type':len})
    pivot_d = pd.pivot_table(dfsjoin_d,index='MSOA11CD',fill_value=0,columns='Crime type',aggfunc={'Crime type':len})
    all_pivot = pivot_r.join(pivot_d, rsuffix="detected").reset_index()
    return all_pivot.fillna(0)

def read_and_add_search(file_path):
    df = pd.read_csv(file_path)
    df = geopandas.GeoDataFrame(df, geometry=geopandas.points_from_xy(df.Longitude, df.Latitude), crs="epsg:4326")
    dfsjoin = geopandas.sjoin(msoas, df) #Spatial join Points to polygons
    pivot_d = dfsjoin.groupby(["MSOA11CD"])['Object of search'].count()

    #pivot_d = pd.pivot_table(dfsjoin,index='MSOA11CD',fill_value=0,columns='Object of search', aggfunc=len)
    
    return pd.DataFrame(pivot_d.fillna(0)).reset_index()

def merge_dfs(list_of_df):
    initial_df = list_of_df[0]
    remaining = list_of_df[1:]
    for pivot in remaining:
        initial_df.groupby("MSOA11CD", as_index=False).sum()
        initial_df = pivot.merge(initial_df.groupby("MSOA11CD", as_index=False).sum(), how="outer")
    #initial_df.columns=["MSOA11CD","Robbery","Detected"]
    print ("dataframe length is " + str(initial_df.shape[0]))
    return initial_df

#we iterate over all files and subfolders in our data directory, and use the appropriate helper functions based on their last characters.
list_of_pivot = []
list_of_searches = []
rootdir = os.getcwd()


for subdir, dirs, files in os.walk("data"):
    for file in files:
        filepath = subdir + os.sep + file

        if filepath.endswith("street.csv"):
            list_of_pivot.append(read_and_add(filepath))
            
        if filepath.endswith("search.csv"):
            list_of_searches.append(read_and_add_search(filepath))

#now we concatenate all our individual dataframes into a single one
print("aggregating robbery")
final_pivot = merge_dfs(list_of_pivot)

print("aggregating searches")
final_search = merge_dfs(list_of_searches)

aggregating robbery

C:\Users\Admin\Anaconda3\envs\geospatial\lib\site-packages\pandas\core\generic.py:3889: PerformanceWarning: dropping on a non-lexsorted multi-index without a level parameter may impact performance.
  obj = obj._drop_axis(labels, axis, level=level, errors=errors)

dataframe length is 1705
aggregating searches
dataframe length is 1953

#finally, merge those into a single pivot with the sum of all incidents during our data, by MSOA.
final_pivot.columns=["MSOA11CD","Robbery","Detected"]
final_pivot = final_pivot.groupby("MSOA11CD").sum()

final_search = final_search.groupby("MSOA11CD").sum()

final_join = final_search.join(final_pivot).reset_index().copy()
final_join.columns=["MSOA11CD", "search_count", "robbery_count","detected_robbery"]

final_join

final_join.sum()

MSOA11CD            E02000001E02000002E02000003E02000004E02000005E...
search_count                                                   477668
robbery_count                                                   60093
detected_robbery                                                 3054
dtype: object

Our final file then, contains nearly 500,000 searches and just over 60,000 robberies (of which a suspect was found in around 3,000) across London’s 938 MSOAs.

msoas["MSOA11CD"] = msoas["MSOA11CD"].astype("string")
final_join["MSOA11CD"] = final_join["MSOA11CD"].astype("string")

msoas = msoas.merge(final_join, on=["MSOA11CD"])

We then re-combine this with our geometry file, before calculating a proportion of robbery detected figure, and a search per robbery rate - the final table is below.

msoas["robbery_solve"] = msoas["detected_robbery"] / msoas["robbery_count"]
msoas["search_rate"] = msoas["search_count"] / msoas["robbery_count"]

msoas.index = msoas["MSOA11NM"]

The data is now processed. Now is probably a good time to write this to a file to retrieve it in the future.

msoas.to_file("msoas.shp")

msoas

Analysis

Let’s start exploring our data. Before doing anything, we’ll use Pysal to create “spatial weights” - this let’s us understand our objects in space, and how they interact. There are a myriad of different methods, but we’ll just pick the 8 nearest MSOAs in space.

# Generate W from the GeoDataFrame
w = weights.distance.KNN.from_dataframe(msoas, k=8)
w.neighbors['City of London 001']

['Islington 023',
 'Southwark 002',
 'Islington 022',
 'Hackney 027',
 'Hackney 026',
 'Southwark 006',
 'Southwark 003',
 'Southwark 009']

# Row-standardization
w.transform = 'R'

We also create a spatial “lag” for our key values - this is one of the most straightforward measures of spatial relationships, and captures the products and weights of our value in neighbouring observations. Essentially, it’s the value of our metric, weighted by the value of the metric in neighbourhing observations - so clusters will expect to be high, while a single high value surrounded by low values will be diminished.

msoas['w_search_count'] = weights.spatial_lag.lag_spatial(w, msoas['search_count'])
msoas['w_robbery_count'] = weights.spatial_lag.lag_spatial(w, msoas['robbery_count'])
msoas['w_rate'] = weights.spatial_lag.lag_spatial(w, msoas['search_rate'])

Let’s start by mapping out our key crime and search figures, and seeing if anything stands out.

f, axs = plt.subplots(nrows=2, ncols=2, figsize=(20, 12))
msoas.plot(column='search_count', 
        cmap='viridis', 
        scheme='quantiles',
        k=5, 
        edgecolor='white', 
        linewidth=0., 
        alpha=0.75, 
        legend=False,
        legend_kwds={"loc": 2},
        ax=axs[0,0]
       )

axs[0,0].title.set_text('Search Count')

msoas.plot(column='robbery_count', 
        cmap='viridis', 
        scheme='quantiles',
        k=5, 
        edgecolor='white', 
        linewidth=0., 
        alpha=0.75, 
        legend=False,
        legend_kwds={"loc": 2},
        ax=axs[0,1]
       )

axs[0,1].title.set_text('Robbery Count')

msoas.plot(column='robbery_solve', 
        cmap='viridis', 
        scheme='quantiles',
        k=5, 
        edgecolor='white', 
        linewidth=0., 
        alpha=0.75, 
        legend=False,
        legend_kwds={"loc": 2},
        ax=axs[1,0]
       )

axs[1,0].title.set_text('Detected Robbery Proportion')

msoas.plot(column='search_rate', 
        cmap='viridis', 
        scheme='quantiles',
        k=5, 
        edgecolor='white', 
        linewidth=0., 
        alpha=0.75, 
        legend=False,
        legend_kwds={"loc": 2},
        ax=axs[1,1]
       )

#contextily pulls a background tile from online providers
axs[1,1].title.set_text('Search per Robbery')
for ax in axs.reshape(-1):
        contextily.add_basemap(ax, 
                           crs=msoas.crs, 
                           source=contextily.providers.Stamen.TerrainBackground)

#ax1.set_axis_off()
#ax2.set_axis_off()

There are a few apparently spatial trends at play: - Searches and robberies seem most common in central London, though searches also seem to cluster on the Western area of the city - The proportion of robberies that are detected seems to be scaterred pretty randomly - The rate of search per robbery seems to form a “U” shape around the soutehrn side of the city, with the Northern edge standing out as potentially “under-policed”

Let’s ignore the spatial dimension for a second, and just create a correlation heatmap. This should let us identify any commonalities between our measured characteristics

corr_cols = ['UsualRes', 'HholdRes', 'ComEstRes', 'PopDen', 'Hholds', 'AvHholdSz',
             'search_count', 'robbery_count',
       'robbery_solve', 'search_rate',
             'w_search_count', 'w_robbery_count','w_rate']

sns.set_theme(style="white")

cmap = sns.diverging_palette(230, 20, as_cmap=True)


plt.figure(figsize=(16, 6))
# define the mask to set the values in the upper triangle to True
mask = np.triu(np.ones_like(msoas[corr_cols].corr(), dtype=np.bool))
heatmap = sns.heatmap(msoas[corr_cols].corr(), mask=mask, vmin=-1, vmax=1, annot=True, cmap=cmap)

The good news is that search and robbery counts are closely correlated: those MSOAs where most robberies occur also see the most searches.

There is also a weak positive correlation (0.3) between commercial households and robberies - this is probably due to some combination of high footfall and affuluence.

There is also a negative relationship between the spatial lag of robberies, and the average household size - I’m not quite sure how to interpret this. Potentially those areas with lots of focused large households are large nexuses of robbery?

Let’s go a bit further by building a scatter plot between robbery and search counts.

sns.scatterplot(x="search_count", y="robbery_count", data=msoas)

It’s a bit of a mess. Most of our robberies are focused on the lower angle, with a few huge outliers. This might benefit from a log transformaton - keeping the key characteristics the same, but normalising our distribution and making the relationship linear.

sns.scatterplot(x=np.log(msoas["search_count"]), y=np.log(msoas["robbery_count"]), data=msoas)

As planned, a lot better - this suggests a linear relationship between both variables, and they should now be normally distributed, allowing us to model them.

sns.displot(x=np.log(msoas["search_rate"]), kde=True)

Spatial Structures

To identify whether geography plays a role in how our values change, we’ll contrast the values to their spatial lag - the local value, impacted by their neighbours. This should smooth out any outliers, and let us get a general value per area, rather than by MSOA.

f, axs = plt.subplots(nrows=3, ncols=2, figsize=(20, 12))
msoas.plot(column='search_count', 
        cmap='viridis', 
        scheme='quantiles',
        k=5, 
        edgecolor='white', 
        linewidth=0., 
        alpha=0.75, 
        legend=False,
        legend_kwds={"loc": 2},
        ax=axs[0,0]
       )

axs[0,0].title.set_text('Search Count')

msoas.plot(column='w_search_count', 
        cmap='viridis', 
        scheme='quantiles',
        k=5, 
        edgecolor='white', 
        linewidth=0., 
        alpha=0.75, 
        legend=False,
        legend_kwds={"loc": 2},
        ax=axs[0,1]
       )

axs[0,1].title.set_text('Spatial Lag - Search Count')

msoas.plot(column='robbery_count', 
        cmap='viridis', 
        scheme='quantiles',
        k=5, 
        edgecolor='white', 
        linewidth=0., 
        alpha=0.75, 
        legend=False,
        legend_kwds={"loc": 2},
        ax=axs[1,0]
       )

axs[1,0].title.set_text('Robbery Count')

msoas.plot(column='w_robbery_count', 
        cmap='viridis', 
        scheme='quantiles',
        k=5, 
        edgecolor='white', 
        linewidth=0., 
        alpha=0.75, 
        legend=False,
        legend_kwds={"loc": 2},
        ax=axs[1,1]
       )

axs[1,1].title.set_text('Spatial Lag - Robbery Count')

msoas.plot(column='search_rate', 
        cmap='viridis', 
        scheme='quantiles',
        k=5, 
        edgecolor='white', 
        linewidth=0., 
        alpha=0.75, 
        legend=False,
        legend_kwds={"loc": 2},
        ax=axs[2,0]
       )

axs[2,0].title.set_text('Search per Robbery')

msoas.plot(column='w_rate', 
        cmap='viridis', 
        scheme='quantiles',
        k=5, 
        edgecolor='white', 
        linewidth=0., 
        alpha=0.75, 
        legend=False,
        legend_kwds={"loc": 2},
        ax=axs[2,1]
       )

axs[2,1].title.set_text('Spatial Lag - Search Rate')
for ax in axs.reshape(-1):
        contextily.add_basemap(ax, 
                           crs=msoas.crs, 
                           source=contextily.providers.Stamen.TerrainBackground)

#ax1.set_axis_off()
#ax2.set_axis_off()

Our spatial weights enable us to clearly identy trends in space: - the search count and robbery count is now far more focused on centralon London, with the exception of a few pockets - the north of London cluster stands out more distinctly as a location where the rate of search per robbery is distinctively lower

Clustering

As an alternative method for exploring our data, we’ll use some machine learning techniques to undertake some clustering. We’ll be using K Means clustering (which focuses on the mean of all our values, and picks the most similar other MSOAs) - given we’re now focusing on modelling, we’ll use the log values.

msoas['ln_search_count'] = np.log(msoas['search_count'])
msoas['ln_robbery_count'] = np.log(msoas['robbery_count'])
msoas['ln_rate'] = np.log(msoas['search_rate'])

msoas['w_search_count'] = weights.spatial_lag.lag_spatial(w, msoas['ln_search_count'])
msoas['w_robbery_count'] = weights.spatial_lag.lag_spatial(w, msoas['w_robbery_count'])
msoas['w_rate'] = weights.spatial_lag.lag_spatial(w, msoas['w_rate'])

w = weights.distance.KNN.from_dataframe(msoas, k=8)

# Row-standardization
w.transform = 'R'

We’ll also include the demographic information provided by MOPAC (household numbers and populations), and use the KMeans Clustering implementation from sklearn. A number of clusters must be specified - I’ve gone for 5 on this occasion. We’ll then map them.

Essentially, we’re trying to aggregate MSOAs into 5 distinct groups, based on our crime and demographic characteristics.

cluster_variables = ['UsualRes', 'HholdRes', 'ComEstRes', 'PopDen', 'Hholds', 'AvHholdSz', 
       'robbery_solve', 'w_search_count', 'w_robbery_count',
       'w_rate', 'ln_search_count', 'ln_robbery_count', 'ln_rate']

# Initialise KMeans instance
kmeans = KMeans(n_clusters=5)

# Set the seed for reproducibility
numpy.random.seed(1234)
# Run K-Means algorithm
k5cls = kmeans.fit(msoas[cluster_variables])

# Assign labels into a column
msoas['k5cls'] = k5cls.labels_
# Setup figure and ax
f, ax = plt.subplots(1, figsize=(14, 9))
# Plot unique values choropleth including a legend and with no boundary lines
msoas.plot(column='k5cls', categorical=True, legend=True, linewidth=0, ax=ax)
# Remove axis
ax.set_axis_off()
# Keep axes proportionate
plt.axis('equal')
# Add title
plt.title(r'Geodemographic Clusters (k-means, $k=5$)')

contextily.add_basemap(ax, crs=msoas.crs, 
                           source=contextily.providers.Stamen.TerrainBackground)
# Display the map
plt.show()

Interestingly, there are no clear patterns here: cluster membership is spread out throughout London. That said, we still don’t know what these actually mean - to understand that, we’ll visualise the distributions of our characteristics in each cluster.

# Index db on cluster ID
tidy_db = msoas.set_index('k5cls')
# Keep only variables used for clustering
tidy_db = tidy_db[cluster_variables]
# Stack column names into a column, obtaining 
# a "long" version of the dataset
tidy_db = tidy_db.stack()
# Take indices into proper columns
tidy_db = tidy_db.reset_index()
# Rename column names
tidy_db = tidy_db.rename(columns={
                        'level_1': 'Attribute', 
                        0: 'Values'})

# Setup the facets
facets = seaborn.FacetGrid(data=tidy_db, col='Attribute', hue='k5cls', \
                  sharey=False, sharex=False, aspect=2, col_wrap=3)
# Build the plot from `sns.kdeplot`
_ = facets.map(seaborn.kdeplot, 'Values', shade=True).add_legend()

The defining metric seems to be density, but some other trends stand out: the densest group (3) also sees the largest proportion of robberies and searches, while conversely group 2 is the very opposite.

Spatial Autocorrelation

We’re now quite happy there are some spatial relationships in our data: where you are in London matters. It appears that an MSOA in the South and North of London might expect differnet numbers of searches and robberies, even were all there other characteristics the same. We can test this by checking for spatial auto-correlation in our data, the primary metric of which is Moran’s I.

moran = esda.moran.Moran(msoas['search_count'], w)

moran.I

0.28837097453822225

Our Moran’s I shows weak, positive autocorrelation accross our entire dataset: an MSOA with a high search count will generally be next to similar MSOAs.

moran.p_sim

0.001

The P statistic suggests this Moran’s I is statistically different to what we would expect if it were random (at P<0.05) - we reject the hypothesis that there is no spatial dimension.

plot_moran(moran);

Regression Modelling

Finally, we’re explore regression models. We’ll build a series of models, each attempting to predict the number of searches in an MSOA, based on the number of robberies and the population density. We’ll start by simple OLS models, and then attempt variations that account for spatial weights.

from pysal.model import spreg
from pysal.lib import weights
from pysal.explore import esda
from scipy import stats
import statsmodels.formula.api as sm
import numpy
import pandas
import geopandas
import matplotlib.pyplot as plt
import seaborn
import patsy

y, X =  patsy.dmatrices("np.log(search_count) ~ np.log(robbery_count) + np.log(PopDen)", msoas, return_type="matrix")

y = np.array(y)
X = np.array(X)

m1 = spreg.OLS(y, X,
                name_y='Log Search Count', name_x=["Constant","Log Robbery Count","Log Pop Density"])

print(m1.summary)

REGRESSION
----------
SUMMARY OF OUTPUT: ORDINARY LEAST SQUARES
-----------------------------------------
Data set            :     unknown
Weights matrix      :        None
Dependent Variable  :Log Search Count                Number of Observations:         983
Mean dependent var  :      5.7488                Number of Variables   :           3
S.D. dependent var  :      0.9163                Degrees of Freedom    :         980
R-squared           :      0.4181
Adjusted R-squared  :      0.4169
Sum squared residual:     479.795                F-statistic           :    352.0632
Sigma-square        :       0.490                Prob(F-statistic)     :  5.987e-116
S.E. of regression  :       0.700                Log likelihood        :   -1042.289
Sigma-square ML     :       0.488                Akaike info criterion :    2090.577
S.E of regression ML:      0.6986                Schwarz criterion     :    2105.249

------------------------------------------------------------------------------------
            Variable     Coefficient       Std.Error     t-Statistic     Probability
------------------------------------------------------------------------------------
            CONSTANT       3.1161071       0.1416924      21.9920574       0.0000000
   Log Robbery Count       0.5845225       0.0260850      22.4083679       0.0000000
     Log Pop Density       0.1141935       0.0357637       3.1929958       0.0014529
------------------------------------------------------------------------------------
Warning: Variable(s) ['Constant'] removed for being constant.

REGRESSION DIAGNOSTICS
MULTICOLLINEARITY CONDITION NUMBER           15.197

TEST ON NORMALITY OF ERRORS
TEST                             DF        VALUE           PROB
Jarque-Bera                       2           4.305           0.1162

DIAGNOSTICS FOR HETEROSKEDASTICITY
RANDOM COEFFICIENTS
TEST                             DF        VALUE           PROB
Breusch-Pagan test                2           4.046           0.1322
Koenker-Bassett test              2           3.543           0.1700
================================ END OF REPORT =====================================

y, X =  patsy.dmatrices("np.log(search_count) ~ np.log(robbery_count) + np.log(PopDen) + np.log(w_robbery_count)", msoas, return_type="matrix")

y = np.array(y)
X = np.array(X)

m2 = spreg.OLS(y, X,
                name_y='Log Search Count', name_x=["Constant","Log Robbery Count","Log Pop Density","Robbery Spatial Weights"])

print(m2.summary)

REGRESSION
----------
SUMMARY OF OUTPUT: ORDINARY LEAST SQUARES
-----------------------------------------
Data set            :     unknown
Weights matrix      :        None
Dependent Variable  :Log Search Count                Number of Observations:         983
Mean dependent var  :      5.7488                Number of Variables   :           4
S.D. dependent var  :      0.9163                Degrees of Freedom    :         979
R-squared           :      0.4188
Adjusted R-squared  :      0.4170
Sum squared residual:     479.214                F-statistic           :    235.1495
Sigma-square        :       0.489                Prob(F-statistic)     :   7.02e-115
S.E. of regression  :       0.700                Log likelihood        :   -1041.693
Sigma-square ML     :       0.488                Akaike info criterion :    2091.386
S.E of regression ML:      0.6982                Schwarz criterion     :    2110.948

------------------------------------------------------------------------------------
            Variable     Coefficient       Std.Error     t-Statistic     Probability
------------------------------------------------------------------------------------
            CONSTANT       2.9955315       0.1797701      16.6631230       0.0000000
   Log Robbery Count       0.5587104       0.0352339      15.8571653       0.0000000
     Log Pop Density       0.0987403       0.0384697       2.5666998       0.0104148
Robbery Spatial Weights       0.0711112       0.0652601       1.0896585       0.2761318
------------------------------------------------------------------------------------
Warning: Variable(s) ['Constant'] removed for being constant.

REGRESSION DIAGNOSTICS
MULTICOLLINEARITY CONDITION NUMBER           27.105

TEST ON NORMALITY OF ERRORS
TEST                             DF        VALUE           PROB
Jarque-Bera                       2           4.293           0.1169

DIAGNOSTICS FOR HETEROSKEDASTICITY
RANDOM COEFFICIENTS
TEST                             DF        VALUE           PROB
Breusch-Pagan test                3           5.101           0.1645
Koenker-Bassett test              3           4.506           0.2118
================================ END OF REPORT =====================================

y, X =  patsy.dmatrices("np.log(search_count) ~ 0 + np.log(robbery_count) + np.log(PopDen)", msoas, return_type="matrix")
y = np.array(y)
X = np.array(X)

m6 = spreg.GM_Error_Het(y, X, 
                           w=w, name_y='Log Search Count', name_x=["Log Robbery Count","Log Pop Density"])
print(m6.summary)

REGRESSION
----------
SUMMARY OF OUTPUT: SPATIALLY WEIGHTED LEAST SQUARES (HET)
---------------------------------------------------------
Data set            :     unknown
Weights matrix      :     unknown
Dependent Variable  :Log Search Count                Number of Observations:         983
Mean dependent var  :      5.7488                Number of Variables   :           3
S.D. dependent var  :      0.9163                Degrees of Freedom    :         980
Pseudo R-squared    :      0.4134
N. of iterations    :           1                Step1c computed       :          No

------------------------------------------------------------------------------------
            Variable     Coefficient       Std.Error     z-Statistic     Probability
------------------------------------------------------------------------------------
            CONSTANT       3.5341582       0.2141206      16.5054581       0.0000000
   Log Robbery Count       0.5823628       0.0365549      15.9311915       0.0000000
     Log Pop Density       0.0124371       0.0397069       0.3132221       0.7541119
              lambda       0.6810127       0.0298280      22.8313537       0.0000000
------------------------------------------------------------------------------------
================================ END OF REPORT =====================================

y, X =  patsy.dmatrices("np.log(search_count) ~ np.log(robbery_count) + np.log(PopDen)", msoas, return_type="matrix")
y = np.array(y)
X = np.array(X)

m7 = spreg.GM_Lag(y, X, w=w, name_y='Log of Search Count', name_x=["Constant","Log Robbery Count","Log Pop Density"])
print(m7.summary)

REGRESSION
----------
SUMMARY OF OUTPUT: SPATIAL TWO STAGE LEAST SQUARES
--------------------------------------------------
Data set            :     unknown
Weights matrix      :     unknown
Dependent Variable  :Log of Search Count                Number of Observations:         983
Mean dependent var  :      5.7488                Number of Variables   :           4
S.D. dependent var  :      0.9163                Degrees of Freedom    :         979
Pseudo R-squared    :      0.4584
Spatial Pseudo R-squared:  0.4175

------------------------------------------------------------------------------------
            Variable     Coefficient       Std.Error     z-Statistic     Probability
------------------------------------------------------------------------------------
            CONSTANT       2.7219978       0.3420296       7.9583682       0.0000000
   Log Robbery Count       0.5614888       0.0311151      18.0455524       0.0000000
     Log Pop Density       0.0818749       0.0430088       1.9036779       0.0569522
W_Log of Search Count       0.1066105       0.0848248       1.2568327       0.2088142
------------------------------------------------------------------------------------
Instrumented: W_Log of Search Count
Instruments: W_Log Pop Density, W_Log Robbery Count
Warning: Variable(s) ['Constant'] removed for being constant.
================================ END OF REPORT =====================================

y, X =  patsy.dmatrices("np.log(search_count) ~ 0 + np.log(robbery_count)", msoas, return_type="matrix")
y = np.array(y)
X = np.array(X)

m7 = spreg.GM_Lag(y, X, w=w, name_y='Log of Search Count', name_x=["Log Robbery Count"])
print(m7.summary)

REGRESSION
----------
SUMMARY OF OUTPUT: SPATIAL TWO STAGE LEAST SQUARES
--------------------------------------------------
Data set            :     unknown
Weights matrix      :     unknown
Dependent Variable  :Log of Search Count                Number of Observations:         983
Mean dependent var  :      5.7488                Number of Variables   :           3
S.D. dependent var  :      0.9163                Degrees of Freedom    :         980
Pseudo R-squared    :      0.4544
Spatial Pseudo R-squared:  0.4130

------------------------------------------------------------------------------------
            Variable     Coefficient       Std.Error     z-Statistic     Probability
------------------------------------------------------------------------------------
            CONSTANT       2.9856130       0.3464531       8.6176549       0.0000000
   Log Robbery Count       0.5879825       0.0320598      18.3401657       0.0000000
W_Log of Search Count       0.1040083       0.0724840       1.4349150       0.1513113
------------------------------------------------------------------------------------
Instrumented: W_Log of Search Count
Instruments: W_Log Robbery Count
================================ END OF REPORT =====================================

Resources I’ve used

• Matt Ashby Crime Mapping course: https://github.com/mpjashby/crimemapping/
• Spatial Modelling for Data Scientists: https://gdsl-ul.github.io/san/
• R for Data Science: https://r4ds.had.co.nz/index.html
• Geocomputation with R: https://geocompr.robinlovelace.net/

Tasks

1. Ingest Data
2. Predict trend by MSOA
3. Quantify MSOA COVID Effect
4. Model

Ingest Data

For this exercise, I’ll be importing crime and robbery data by MSOA.MSOAs are geographical units specifically designed for analysis, and to be comparable: they all have an average population of just over 8,000. There is a compromise here between smaller geographical units (that create more variance that may help us identify predictors), but the necessity for enough crime per unit to identify meaningful trends - MSOAs should be suitable.

To build our process, we’ll start by taking one month of crime data, exploring it, and writing all our steps for automation.

test_df <- read.csv("crimes/2018-01/2018-01-metropolitan-street.csv")

Our crime data is categorised according to the Home Office major crime types, and like Python, we can list them all through the “unique” function. Here I’ll be focusing on robbery and burglary: two crime types that are heavily reliant on encountering victim’s in public spaces, and as such should be affected by the “COVID effect”.

To avoid this getting particularly computationally intensive, let’s write a function to pull out robberies and burglaries, and assign them a specific MSOA. Then we can iterate over all our months and get monthly counts for each offence type.

subset_df <- filter(test_df, Crime.type=="Burglary" | Crime.type=="Robbery")
head(subset_df)