By Tara Hoke
With the proliferation of artificial intelligence-powered tools in today’s engineering workforce, a case study relating to the downfall of IBM’s $4 billion health care AI system offers a message of caution for engineering professionals when engaging with new technologies.
Situation
Long-time followers of the television quiz show Jeopardy are likely to recall watching the program in February 2011, when IBM’s machine-learning system Watson defeated two of the show’s greatest human champions over a series of three episodes. While the event was designed to offer a public demonstration of the industry giant’s growing capabilities in natural language processing, it also gave a major boost to IBM’s efforts to bring AI-powered tools into the commercial marketplace.
In 2015, IBM announced the launch of Watson for Oncology, an AI platform its creators boldly claimed would revolutionize the care of cancer patients. Trained on massive amounts of patient records, case reports, medical papers, and other data, Watson for Oncology was designed to provide new insights to clinicians in diagnosing and treating even the rarest and most complex cases.
To build this new platform, IBM sank a reported $4 billion into the acquisition of firms offering health care data and analytic capabilities. It also reached a partnership agreement with one of the United States’ most prestigious cancer treatment hospitals, whose team of clinicians would train the AI tool to “think like a doctor.”
Yet the doctors working to train the Watson for Oncology system reportedly found that using real case reports was “messier” than expected. Doctors’ notes were often difficult to read or transcribe; case reports, gene sequencing, and other patient data were incomplete or scattered across multiple, sometimes incompatible, digital platforms.
Even complete patient profiles proved to be of questionable value due to the pace of changes in treatment guidelines. To resolve these challenges, the team of trainers increasingly turned to “synthetic” case studies: hypothetical patient data used as the basis to input diagnoses and treatment protocols.
Unfortunately, this change meant that, instead of representing the collective wisdom of doctors worldwide, the Watson for Oncology system was largely trained based on cases involving the recommendations of a handful of doctors at a single (albeit prestigious) U.S. hospital. Users of the new tool complained that the system’s recommendations did not account for differences in physician preferences or local clinical practices, nor could they be generalized to all cases.
These issues were only magnified for the system’s global users. IBM had marketed the tool as a means of reducing inequities in access to health care, yet all too often the system proposed recommendations that could not be implemented in parts of the world with differences in drug availability or required treatment protocols.
Worse yet, in 2017, media sources reported that internal testing by IBM found “multiple examples of unsafe and incorrect treatment recommendations” from the system. For example, in a test involving a hypothetical 65-year-old man with lung cancer and a history of severe bleeding, the system prescribed bevacizumab, which is a drug associated with an increased risk of severe or deadly hemorrhage. While IBM dismissed these errors as part of the training process and not reflective of the platform’s true performance in practice, the negative publicity proved harmful to the already beleaguered tool.
Perhaps the greatest source of criticism for the Watson for Oncology tool, though, was its failure to live up to its own hype. While many customers found it useful as a research aid, there was little evidence that Watson for Oncology offered any measurable improvement in clinical decision-making or patient outcomes. Far from being a source of new insights or advancements, it seemed the platform’s value was primarily limited to confirming treatment plans or diagnoses a clinician had already made.
With sales flagging and several high-profile customers signaling their intent to look elsewhere for AI solutions, by 2019 IBM had begun to scale back on its support of the Watson for Oncology platform. Finally, in 2023, IBM ended years of speculation about the platform’s prospects by announcing that it was exiting the health care business. Watson’s health care data and analytics assets were sold to a private equity company for an undisclosed amount, leading more than one major technology reporter to quip that the AI tool had been “sold for parts.”
Question
What ethical lessons does this case study offer to the engineering profession?
Discussion
While civil engineering practice is constantly being reshaped by new techniques and innovations, perhaps no advancement in recent memory has elicited as much excitement — or fear — as AI. Whether it is viewed as a technological marvel that will raise human achievement to unprecedented heights or a data-crunching juggernaut that will replace vast segments of the workforce, few seem to have any doubt of AI’s transformational effect on the field of civil engineering.
Whatever the future may hold, it is clear that AI is already a powerful new tool for the engineering profession, but like all such tools, it carries with it an ethical obligation for engineers to investigate and understand its range of uses.
Section 1h of the ASCE Code of Ethics instructs engineers to “consider the capabilities, limitations, and implications of current and emerging technologies when part of their work.” This language serves a dual purpose: It reminds engineers to embrace developments that may improve the delivery of engineering services, while at the same time it recommends caution with unfamiliar or untested tools.
This case study demonstrates many of the risks surrounding AI, including concerns about the integrity of data used in training AI systems, the ease of introducing unintended biases, and an overall lack of transparency about the quality or verifiability of AI outputs. Other concerns not directly referenced in this case include data security, privacy, and legal or ethical questions about the ownership of AI-created content.
In view of these concerns, misuse of AI tools may pose a multitude of risks to an engineering professional. Overreliance on AI may entice an engineer to exceed his or her technical competence, may result in subpar service to a client or employer, or may lead to careless or ill-informed decisions that threaten the health or safety of persons who depend on the engineer’s services. The choice to carefully consider AI’s capabilities, limitations, and implications is thus not merely a wise business practice, but also an integral piece of the engineer’s ethical duty to others.
On July 18, 2024, ASCE’s Board of Direction adopted ASCE Policy Statement 573 — Artificial intelligence and engineering responsibility. Echoing the themes discussed here, this policy states as follows:
AI has the potential to enhance efficiency, innovation, and sustainability in civil engineering practices. ... However, AI cannot be held accountable, nor can it replace the training, experience, and (judgment) of a professional engineer in the planning, designing, building, and operation of civil engineering projects and the protection of the public health, safety, and welfare.
Despite the faults of the Watson for Oncology platform, it is heartening to note that there are no reports of the system causing harm to patients. It seems clear that the practitioners who used the tool were careful to always balance Watson’s guidance against their own professional judgment, not relying on the technology to the detriment of their ethical obligations or the individuals they served. This example is also noteworthy, then, as a model of ethical behavior for engineers engaging with AI or with any other innovation.
Tara Hoke is ASCE’s general counsel and a contributing editor to Civil Engineering.
This article first appeared in the March/April 2025 print issue of Civil Engineering as "The Ethics of Artificial Intelligence."
