I'm specifically interested in tying doctors to their published papers. The key issue is that using name alone will result many collisions. I'm wondering what set of features I would need to reliably connect a doctor with a given published paper? Aside from weak features such as specialty, is there any database that has a link between doctor NPI and papers published?
I've seen this on linking NPIs to authors in PubMed but it seems rather unreliable.
It's not an easy question to answer even for groups with a lot of leverage such as Research Gate (RG). RG has it's own (I assume proprietary) author matching algorithm which has caused problems in the past.
They use the name of the author (in different combinations) to suggest authorship to RG users (so has you said, it does causes a lot of problems). Every once in a while users that are not the author accept the suggestions and, from the portal point of view, gain the equivalent reputation. It's a serious business that requires quite a bit of R&D before making a decision.
That being said I can't answer with certainties only with reasonable possibilities. A few questions I would make (and hope to answer with a bit of data analysis):
What is the probability that an author with a confirmed publication in an identified journal will publish in that journal again? - Journal Name
What is the probability that an author that has partnered with other authors will repeat the same co-authoring combination? Co-author Names
What is the expected time story for publishing for each author? Authors rarely publish articles 20 years apart. Typically they publish more and more, or less and less. Time frame
How frequently do authors change the institution they belong to? Institution Name
What are the preferred keywords for a given author? Key Names
What are the preferred citations made by a given author? Bibliography
All of the questions above require quite a bit of Text Mining and String Matching, as well as solid dataset to start your mining.
Some publishers have their own API although I can't say much about permissiveness (never tried it myself). RG has been promising one for years but, as far as I know, it still does not exist.
An unlikely thing I remember now is the inspiring story of Aaron Swartz. This activist, along with other persons, successfully managed to create large open archives for books and articles. Should that information still exist it might be worth your time to take a look there.
Also if you have a list of the journals you are considering (you've mentioned only "doctors" which is a bit vague) you can try and see with the publisher if they have any way of accessing their database.
