<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing DTD v2.0 20040830//EN" "http://dtd.nlm.nih.gov/publishing/2.0/journalpublishing.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink" article-type="letter" dtd-version="2.0">
  <front>
    <journal-meta>
      <journal-id journal-id-type="publisher-id">JMIR</journal-id>
      <journal-id journal-id-type="nlm-ta">J Med Internet Res</journal-id>
      <journal-title>Journal of Medical Internet Research</journal-title>
      <issn pub-type="epub">1438-8871</issn>
      <publisher>
        <publisher-name>JMIR Publications</publisher-name>
        <publisher-loc>Toronto, Canada</publisher-loc>
      </publisher>
    </journal-meta>
    <article-meta>
      <article-id pub-id-type="publisher-id">v23i6e29336</article-id>
      <article-id pub-id-type="pmid">34132643</article-id>
      <article-id pub-id-type="doi">10.2196/29336</article-id>
      <article-categories>
        <subj-group subj-group-type="heading">
          <subject>Letter to the Editor</subject>
        </subj-group>
        <subj-group subj-group-type="article-type">
          <subject>Letter to the Editor</subject>
        </subj-group>
      </article-categories>
      <title-group>
        <article-title>Author's Reply to: Periodic Manual Algorithm Updates and Generalizability: A Developer’s Response. Comment on “Evaluation of Four Artificial Intelligence–Assisted Self-Diagnosis Apps on Three Diagnoses: Two-Year Follow-Up Study”</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="editor">
          <name>
            <surname>Derrick</surname>
            <given-names>Thomas</given-names>
          </name>
        </contrib>
      </contrib-group>
      <contrib-group>
        <contrib id="contrib1" contrib-type="author" corresp="yes">
          <name name-style="western">
            <surname>Ćirković</surname>
            <given-names>Aleksandar</given-names>
          </name>
          <degrees>MD, MSc</degrees>
          <address>
            <addr-line>Schulgasse 21</addr-line>
            <addr-line>Weiden, 92637</addr-line>
            <country>Germany</country>
            <phone>49 1788603753</phone>
            <email>aleksandar.cirkovic@mailbox.org</email>
          </address>
          <ext-link ext-link-type="orcid">https://orcid.org/0000-0002-2122-5175</ext-link>
        </contrib>
      </contrib-group>
      <author-notes>
        <corresp>Corresponding Author: Aleksandar Ćirković <email>aleksandar.cirkovic@mailbox.org</email></corresp>
      </author-notes>
      <pub-date pub-type="collection">
        <month>6</month>
        <year>2021</year>
      </pub-date>
      <pub-date pub-type="epub">
        <day>16</day>
        <month>6</month>
        <year>2021</year>
      </pub-date>
      <volume>23</volume>
      <issue>6</issue>
      <elocation-id>e29336</elocation-id>
      <history>
        <date date-type="received">
          <day>2</day>
          <month>4</month>
          <year>2021</year>
        </date>
        <date date-type="accepted">
          <day>13</day>
          <month>5</month>
          <year>2021</year>
        </date>
      </history>
      <copyright-statement>©Aleksandar Ćirković. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 16.06.2021.</copyright-statement>
      <copyright-year>2021</copyright-year>
      <license license-type="open-access" xlink:href="https://creativecommons.org/licenses/by/4.0/">
        <p>This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.</p>
      </license>
      <self-uri xlink:href="https://www.jmir.org/2021/6/e29336" xlink:type="simple"/>
      <related-article related-article-type="commentary-article" id="v22i12e18097" ext-link-type="doi" xlink:href="10.2196/18097" vol="22" page="e18097" xlink:type="simple">https://www.jmir.org/2020/12/e18097</related-article>
      <related-article related-article-type="commentary-article" id="v23i6e26514" ext-link-type="doi" xlink:href="10.2196/26514" vol="23" page="e26514" xlink:type="simple">https://www.jmir.org/2021/6/e26514</related-article>
      <kwd-group>
        <kwd>artificial intelligence</kwd>
        <kwd>machine learning</kwd>
        <kwd>mobile apps</kwd>
        <kwd>medical diagnosis</kwd>
        <kwd>mHealth</kwd>
        <kwd>symptom assessment</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <p>I would like to thank Gilbert et al [<xref ref-type="bibr" rid="ref1">1</xref>] for their useful contributions and tips to help improve our work [<xref ref-type="bibr" rid="ref2">2</xref>].</p>
    <p>Regarding their first point of critique, it is true that the evaluation lacked the technique to determine with certainty whether the algorithms used were or were not “locked” according to the terminology used by the Food and Drug Administration. For that reason, I stated in the <italic>Discussion</italic> section that some details of the study results only <italic>suggest</italic> a nonlocked direction [<xref ref-type="bibr" rid="ref2">2</xref>]. Unfortunately, the abstract lacks this softening term. It remains true, however, that for regulators and physicians to correctly assess the functions of such software, either the algorithms would have to be disclosed—which might not be in the interest of the competing companies—or advanced testing models would have to be developed, possibly leading to a “cat-and-mouse” game similar to other regulatory fields. In developed countries where legal liabilities have to be clearly distributed and delimited, neither responsible physicians nor authorities will want to rely on mere published statements of company spokespeople alone—hard data from unbiased sources will be needed in the future.</p>
    <p>The subsequent publication of the Ada study of a 200-vignette assessment is highly appreciated, as herein the authors were able to evaluate a considerably larger amount of data [<xref ref-type="bibr" rid="ref3">3</xref>]. As I had stated, with a small sample size as a limitation, it does remain a possibility that the apps’ poor results as demonstrated in our study are within data variance norms or due to our own bias. However, the results did capture an experience that any potential user or patient with eye problems could have possibly encountered in the same way. Regulations will hopefully minimize risks for all users. Only thorough investigations, including but not limited to manufacturers’ evaluations, will help us better understand the effects the apps will have on public health. Until then, our knowledge base will consist of various analyses with possibly conflicting results that we will have to make sense of.</p>
  </body>
  <back>
    <app-group/>
    <fn-group>
      <fn fn-type="con">
        <p>The author AC is currently not affiliated with any institution, but is an Independent Scholar.</p>
      </fn>
      <fn fn-type="conflict">
        <p>None declared.</p>
      </fn>
    </fn-group>
    <ref-list>
      <ref id="ref1">
        <label>1</label>
        <nlm-citation citation-type="journal">
          <person-group person-group-type="author">
            <name name-style="western">
              <surname>Gilbert</surname>
              <given-names>S</given-names>
            </name>
            <name name-style="western">
              <surname>Fenech</surname>
              <given-names>M</given-names>
            </name>
            <name name-style="western">
              <surname>Idris</surname>
              <given-names>A</given-names>
            </name>
            <name name-style="western">
              <surname>Türk</surname>
              <given-names>E</given-names>
            </name>
          </person-group>
          <article-title>Periodic Manual Algorithm Updates and Generalizability: A Developer’s Response. Comment on “Evaluation of Four Artificial Intelligence–Assisted Self-Diagnosis Apps on Three Diagnoses: Two-Year Follow-Up Study”</article-title>
          <source>J Med Internet Res</source>
          <year>2021</year>
          <month>06</month>
          <volume>23</volume>
          <issue>6</issue>
          <fpage>e26514</fpage>
          <comment>
            <ext-link ext-link-type="uri" xlink:type="simple" xlink:href="https://www.jmir.org/2021/6/e26514"/>
          </comment>
          <pub-id pub-id-type="doi">10.2169/26514</pub-id>
        </nlm-citation>
      </ref>
      <ref id="ref2">
        <label>2</label>
        <nlm-citation citation-type="journal">
          <person-group person-group-type="author">
            <name name-style="western">
              <surname>Ćirković</surname>
              <given-names>Aleksandar</given-names>
            </name>
          </person-group>
          <article-title>Evaluation of Four Artificial Intelligence-Assisted Self-Diagnosis Apps on Three Diagnoses: Two-Year Follow-Up Study</article-title>
          <source>J Med Internet Res</source>
          <year>2020</year>
          <month>12</month>
          <day>04</day>
          <volume>22</volume>
          <issue>12</issue>
          <fpage>e18097</fpage>
          <comment>
            <ext-link ext-link-type="uri" xlink:type="simple" xlink:href="https://www.jmir.org/2020/12/e18097/"/>
          </comment>
          <pub-id pub-id-type="doi">10.2196/18097</pub-id>
          <pub-id pub-id-type="medline">33275113</pub-id>
          <pub-id pub-id-type="pii">v22i12e18097</pub-id>
          <pub-id pub-id-type="pmcid">PMC7748958</pub-id>
        </nlm-citation>
      </ref>
      <ref id="ref3">
        <label>3</label>
        <nlm-citation citation-type="journal">
          <person-group person-group-type="author">
            <name name-style="western">
              <surname>Gilbert</surname>
              <given-names>S</given-names>
            </name>
            <name name-style="western">
              <surname>Mehl</surname>
              <given-names>A</given-names>
            </name>
            <name name-style="western">
              <surname>Baluch</surname>
              <given-names>A</given-names>
            </name>
            <name name-style="western">
              <surname>Cawley</surname>
              <given-names>Caoimhe</given-names>
            </name>
            <name name-style="western">
              <surname>Challiner</surname>
              <given-names>Jean</given-names>
            </name>
            <name name-style="western">
              <surname>Fraser</surname>
              <given-names>Hamish</given-names>
            </name>
            <name name-style="western">
              <surname>Millen</surname>
              <given-names>Elizabeth</given-names>
            </name>
            <name name-style="western">
              <surname>Montazeri</surname>
              <given-names>Maryam</given-names>
            </name>
            <name name-style="western">
              <surname>Multmeier</surname>
              <given-names>Jan</given-names>
            </name>
            <name name-style="western">
              <surname>Pick</surname>
              <given-names>Fiona</given-names>
            </name>
            <name name-style="western">
              <surname>Richter</surname>
              <given-names>Claudia</given-names>
            </name>
            <name name-style="western">
              <surname>Türk</surname>
              <given-names>Ewelina</given-names>
            </name>
            <name name-style="western">
              <surname>Upadhyay</surname>
              <given-names>Shubhanan</given-names>
            </name>
            <name name-style="western">
              <surname>Virani</surname>
              <given-names>Vishaal</given-names>
            </name>
            <name name-style="western">
              <surname>Vona</surname>
              <given-names>Nicola</given-names>
            </name>
            <name name-style="western">
              <surname>Wicks</surname>
              <given-names>Paul</given-names>
            </name>
            <name name-style="western">
              <surname>Novorol</surname>
              <given-names>Claire</given-names>
            </name>
          </person-group>
          <article-title>How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs</article-title>
          <source>BMJ Open</source>
          <year>2020</year>
          <month>12</month>
          <day>16</day>
          <volume>10</volume>
          <issue>12</issue>
          <fpage>e040269</fpage>
          <comment>
            <ext-link ext-link-type="uri" xlink:type="simple" xlink:href="https://bmjopen.bmj.com/lookup/pmidlookup?view=long&#38;pmid=33328258"/>
          </comment>
          <pub-id pub-id-type="doi">10.1136/bmjopen-2020-040269</pub-id>
          <pub-id pub-id-type="medline">33328258</pub-id>
          <pub-id pub-id-type="pii">bmjopen-2020-040269</pub-id>
          <pub-id pub-id-type="pmcid">PMC7745523</pub-id>
        </nlm-citation>
      </ref>
    </ref-list>
  </back>
</article>
