The "Father of History", Herodotus, might be an interesting case study for this particular question.
Ancient historians have often taken Herodotus as a valuable source through which they can interpret Egyptian and Babylonian culture and history, but in recent years a few scholars have revived doubts of some of the earlier scholars who suspected
Histories was a narrative more fictional than historical.
The work reads as if its narrator wants us to believe he really did travel to all those places to garner first hand information etc, and it frequently claims to cite actual inscriptions on stone as well as report what various authorities personally told him.
On occasion Herodotus even confesses that he has encountered opposing narratives about a particular event so he sets both of them out for the reader -- and will sometimes explain his own reasons for preferring one over the other.
That all sounds very genuine as "historical" writing.
But problems or at least questions arise when scholars examine the text as literature -- or do a literary analysis on it.
Mandell and Freedman, for instance, demonstrate some remarkable similarities between the Histories of
Herodotus and the "Primary History" (Genesis to 2 Kings) of our Bible. Both works can be read as theological narratives demonstrating how the will of the deity is worked out through a historical-sounding narrative.
Nielsen also addresses strong similarities between the work of Herodotus and the Deuternomist's biblical narrative.
Then we meet the "minimalists" -- Thompson, Lemche, Davies, Whitelam, et al -- who ask us to reconsider the evidence for any historical validity at all to the history of "biblical Israel" at least up to the time of the sixth century bce.
Before these guys made serious ripples, however, we already had scholars like
Fehling who opened up the case for Herodotus likewise being in very large measure
fiction. Herodotus never travelled as he claimed; his supposed quotations of stone inscriptions were fabricated; his assurances that various local authorities told him stories were worthless. Armayor and a succession of other scholar appear to have published more granular criticisms of Herodotus in Fehling's wake.
What has opened up Herodotus to a case for fabrication, fiction, is literary analysis coupled with a bit of testing against independent third-party witnesses.
(I do acknowledge that not all ancient historians have followed Fehling, Armayor and others any more than biblical scholars have rushed en masse to follow "the minimalists". Who wants to admit to a method and conclusions that undermine a life-time's work?)
Literary analysis can show the rhetorical functions of many of Herodotus's seemingly factual claims. Many of them function too neatly as narrative persuaders -- far more-so than we encounter in other less indisputable historical works. Moreover, comparisons can be made with other works either known to Herodotus and/or from his own times, and one can reasonably infer that Herodotus has drawn upon literary sources rather than first-hand accounts from priests in Egypt. And in at least one instance we can see with our own eyes that Herodotus has failed to quote an inscription on a stone monument in Asia Minor with reasonable accuracy.
The point here is, I think, not that we cannot distinguish between history and fiction, but rather that we in fact can distinguish between them.
Literary analysis, including comparative literary studies, along with testing against independent sources -- that's a good start.