When Bruce and I were initially discussing what Halcyon would become, the question of what data and information to prefer in our ingestion processes, at first, was important. Perhaps I did not help matters much when Bruce asked me “what information do you want?” and I responded “all of it.” 

That answer was not meant to be flip, really. Halcyon absolutely does want to ingest, format, de-duplicate, tag, and format every bit of information that helps energy transition and decarbonization professionals make better, and faster, business decisions. And we particularly want the information that is hard to deal with today: dockets, integrated resource plans, project development milestones, narrowly constructed regulatory filings, comments on proposed rulemaking, and the like. 

But it is important to know, and to note, that the process of building an information catalog and making it maximally useful actually is a highly selective process with rigorous controls. It is not so much “hoover up all of the information” that is useful, as much as it is “collect the right information, rigorously sourced, and processed fit for purpose.” 

Halcyon is currently ingesting thousands of documents a week, up to hundreds of thousands a week when we initiate a new data collector from a new source. The result of this process is not only clean — in that information has been deliberately processed into a format that can be queried — but also is limited to precise sources. To put it another way: every document published by the PJM Interconnection is useful and part of a coherent whole, but every document published about the PJM interconnection is not.  

We describe information that is ingested and processed in this fashion as authoritative. The source is known and trusted, and so too is the way in which we incorporate it into our information system. We benefit, in other words, from having less information at first, not more. 

This process is an edit of sources, but not an edit within those sources. Once that initial authoritativeness has been established, everything is initially relevant, and equally valid for querying. Users then determine what is most specifically relevant to their purposes and needs. To return to the PJM interconnection example: anything from PJM itself is authoritative information, but it is our users who will determine which information is specifically relevant to their needs and purposes.  

Querying authoritative information corpora is, in effect, constraining searches to only the most relevant sources. Our founding engineer Alexander Huras calls this “narrowing the map.” Limiting scope is more effective than wading through the entirety of the internet. It is information that customers want, not the internet, and the paradox therein is that constraining a search to a purposefully bounded set of information is more powerful than searching through everything, everywhere. 

It is important to note that constraints do not mean limitations. Constraints allow Halcyon to expand our authoritative information sources knowing that we have scalable, replicable processes for doing so. And as we build a platform that learns from our existing ingestion and formatting processes, we expect  that each new authoritative source integrates with everything that comes before it. As more professionals use our platform, multiple domains of authority will emerge from these connections and become more useful over time - this is the flywheel. Constraints mean that connections between new information sources are themselves authoritative, and more meaningful in the process.  

A platform that learns from users is also a platform that solves problems that users care about. That’s a matter not so much of models and product offerings as it is understanding what, exactly, needs doing. It’s worth repeating a thought from our July post r>g; Specialization in the age of AI: 

"That brings us to the crux of the matter: whether or not a generalist model can compete with a specialist solution is less important — and less valuable — than figuring out exactly what problem to solve, exactly what product to build to help solve it, and exactly how best to monetize that product."

Infinite information scope is not actually helpful for accelerating better energy transition energy decisions. Infinite connection between constrained, authoritative information sets, integrated into a platform that learns from its users and guides them at the same time, is meaningful. Once established, these connections can be used for business development, lead generation, and much more.

Comments or questions? We’d love to hear from you - sayhi@halcyon.eco, or find us on LinkedIn and Twitter