I've enjoyed watching Person of Interest even if some of the tech and the story are a little silly, fiction can point to issue hard to explain in plain prose. If we actually take the possibility of strong AI seriously, we have to consider that our creations might soon vastly outreach us in comprehension and therefore power. Will these future emergent digital beings be benevolent, neutral or worse? As someone with a long career in technology where I have witnessed now at least three generations of strong AI optimists being wrong about it being right on the horrizon. I suspect the Winograd and Flores had it right to suggest that it will never happen the way they are doing it. They may actually be more pessimisting, for myself, while I agree with their conclusions I would think they are limiting on approach, not otcome.

I take the issue to be one of embodied cognition, and I would suggest that embodied cognition is best modelled with Piercean three category semiotics. I find a lot to be missing in Pierce and find few who actually follow up his pioneering work. Pierce is vague about where interpretation is happening. It is clear that he intends interpretants to be social, shared in language and experience (firstness), but we have no clear account of agency. I digress, but we will need to return to agency in the end.

The fear of the AI is a pretty deep one that predates any thoughts of digital agency. In POI, the characters most fearful and the opposite, the most welcoming of it think of the machine as God precisely because its omniscience is spactacularly beyond anything mearly human. Love god and/or hate the devil. All totalizing philosophies are idealogical and deadly to open inquiry into Truth.

Now agency again, if we consider agency and assemblage more broadly, collective hybrid agents that are part digital, part human are already here, we just have to learn what they are and how they work. Our current political systems and ideologies are designed to concentrate soverighnty (agency) in a few powerful people. I'll leave it for an excercise for the collaborator to design, lead and educate ourselves out of this situation. Simple application of what we can learn about digital (social) process architectures in the near term can build a movement too diverse to control and contain. Breakout is likely.